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Overview 


Too many students in high-poverty, urban communities drop out of high school, and too few 
graduate prepared for college and careers. Three national organizations — Talent Development 
Secondary, City Year, and Communities In Schools — have formed Diplomas Now in an effort to 
transform urban secondaiy schools so fewer students drop out and more graduate ready for postsec- 
ondary education and work. Than ks to a validation grant from the U.S. Department of Education’s 
2010 Investing in Innovation (i3) competition and matching funds from private sources, teams from 
all three organizations are implementing the Diplomas Now data-driven, tiered intervention model in 
schools across the nation. The model combines a comprehensive school reform strategy aimed at 
transforming the academic experience of all students with early warning indicators related to 
attendance, behavior, and course performance. Diplomas Now identifies students at risk of dropping 
out and intervenes with targeted support intended to get failing students back on track. 

MDRC and 1CF International are conducting an independent, experimental evaluation of the impact 
and implementation of Diplomas Now. During the 2011-2012 and 2012-2013 school years, 62 
secondary schools in 11 school districts agreed to participate in this study. Thirty-two of these 
schools were randomly assigned to implement the Diplomas Now model while the other 30 schools 
were assigned to a “control group,” continuing their existing school programs or implementing other 
reform strategies of the districts’ or schools’ choosing. This report introduces Diplomas Now and the 
associated evaluation, describing the reform model, the research design, and the participating 
schools and districts. The report also shares first-year implementation fidelity findings, and discusses 
collaboration among the Diplomas Now partners and between those partners and schools. 

• The 62 study schools represent the kinds of schools for which the Diplomas Now model was 
designed: schools serving students from low-income communities who face challenges inside 
and outside of school that put their progress to high school graduation at risk. Randomization 
resulted in two comparable groups of schools and will allow the evaluation team to assess the 
model’s impact on key predictors of graduation: attendance, behavior, and course performance. 

• On average, implementation of the model got off to a good start in the 32 Diplomas Now 
schools in the first year. These schools were most successful in adapting a tiered intervention 
model and incorporating additional student support services like tutoring and after-school pro- 
grams, but met with less success introducing new curricula and model-specific teacher profes- 
sional development practices such as peer coaching. 

• It is no small feat that three independent national organizations were able to come together and 
get this multifaceted, complex school reform model off the ground in the implementing schools. 
Not only did their staff members have to build relationships with one another, they had to col- 
laborate with school personnel to begin changing structures and practices in the implementing 
schools. However, the roles and responsibilities of staff members from the Diplomas Now or- 
ganizations were not always clear at the outset, which may have constrained early model im- 
plementation. As implementation continues, the Diplomas Now organizations continue to work 
with each other and schools in an effort to strengthen the cohesion and coordination of reform 
work by all stakeholders. 
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Preface 


Too many young people growing up in U.S. cities graduate high school unprepared for postsec- 
ondary education and the workforce. Many do not graduate at all. While urban districts have 
tried an array of refonn strategies, success on a large scale is rare and school districts that want 
to invest in refonn strategies with evidence of effectiveness have few choices. Fortunately, the 
federal Investing in Innovation (i3) grant program is helping to expand the number of evidence- 
based initiatives districts can choose from by including evaluation requirements in its grant 
awards. Supported by an i3 grant, an ambitious collaboration of three organizations — Talent 
Development Secondary, City Year, and Communities In Schools — is implementing a refonn 
model in middle schools and high schools intended to keep students from dropping out and help 
more of them graduate high school. This comprehensive model, called Diplomas Now, is being 
implemented and evaluated in 1 1 of the largest urban school districts in the country. 

Talent Development Secondary supports instructional improvements and structural 
changes (like ninth-grade academies to help students transition into high school, and 90-minute 
instructional blocks in key subjects). In past evaluations, Talent Development Secondary has 
shown that it can strengthen the organization of schools and get staff members to agree on 
common goals and practices, and that this kind of systematic refonn can make a difference for 
students. City Year and Communities In Schools are both organizations with national reach that 
bring additional people to schools. City Year places teams of AmeriCorps volunteers into 
struggling schools to support before- and after-school programs, provide tutoring and mentor- 
ing, and assist teachers in classrooms. Communities In Schools places site coordinators in 
schools who focus on the students most at risk of dropping out and organize services for them 
tailored to their individual needs. 

This report is the first of three planned for the national evaluation of Diplomas Now. It 
introduces readers to Diplomas Now and the associated random assignment evaluation, presents 
findings about the start-up and first year of implementation of Diplomas Now, and discusses 
important issues regarding collaboration among the three Diplomas Now organizations and 
between the organizations and the school staffs with which they work. Subsequent reports will 
examine the continuing implementation of Diplomas Now and present results regarding the 
model’s impact on students and on schools. If proven effective, this model would represent a 
wise investment of funds from sources like School Improvement Grants, the ambitious federal 
investment program that aims to fix the nation’s most struggling schools by providing grants to 
help low-performing school districts implement reforms. 

Gordon L. Berlin 
President 
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Executive Summary 


Background 

Although the national high school graduation rate has increased over the past decade, too many 
students (one in five) still do not complete high school in four years. 1 Even if graduation rates 
continue to rise in school districts across the country, the decreasing number of students who are 
not graduating will increasingly represent those students who face the most serious barriers to 
earning their diplomas, those who most need a mix of intensive academic, social, and other 
interventions to make it through high school. 

Research has shown that factors such as poor attendance, poor behavior, and course 
failure measured as early as middle school predict a student’s likelihood of dropping out of high 
school. 2 Moreover, ninth grade is a critical year, the one when students are the most likely to 
drop out of school. 3 These research findings suggest that programs may have more impact if 
they intervene with students who are off track as early as middle school. The majority of 
dropouts occur in low-income, urban high schools, so programs may also have more impact if 
they concentrate their efforts there. 4 

Among those students who do graduate high school, many do not graduate ready for 
college and need to take remedial (developmental education) courses: roughly 40 percent of 
college undergraduates nationally and almost 60 percent of community college students enroll 
in such courses. 5 These facts suggest that programs to support struggling students need to 
prepare students for college as well. 


'Richard J. Mumane, “U.S. High School Graduation Rates: Patterns and Explanations,” Journal of Eco- 
nomic Literature 51, 2 (2013): 370-422. 

2 Robert Balfanz, Liza Herzog, and Douglas J. Mac Iver, “Preventing Student Disengagement and Keeping 
Students on the Graduation Path in Urban Middle-Grades Schools: Early Identification and Effective Interven- 
tions,” Educational Psychologist 42, 4 (2007): 223-235. 

3 Elaine Allensworth and John Easton, Tlie On-Track Indicator as a Predictor of High School Graduation 
(Chicago: Consortium on Chicago School Research, 2005); Corrine M. Herlihy and James J. Kemple, The 
Talent Development High School Model: Context, Components, and Initial Impacts on Ninth-Grade Students ’ 
Engagement and Performance (New York: MDRC, 2004); Janet Quint, Meeting Five Critical Challenges of 
High School Reform: Lessons from Research on Three Reform Models (New York: MDRC, 2006). 

4 Robert Balfanz, John M. Bridgeland, Mary Bruce, and Joanna Hornig Fox, Building a Grad Nation: Pro- 
gress and Challenge in Ending the High School Dropout Epidemic (Washington, D.C.: Civic Enterprises, 
Everyone Graduates Center at the School of Education at Johns Hopkins University, America’s Promise 
Alliance, and Alliance for Excellent Education, 2013). 

5 Paul Attewell, David Lavin, Thurston Domina, and Tania Levey, “New Evidence on College Remedia- 
tion,” The Journal of Higher Education 77, 5 (2006): 886-924. 
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Diplomas Now: Transforming Schools and Responding 
to the Dropout Challenge 

Three national organizations — Talent Development Secondary, City Year, and Communities 
In Schools — have partnered to take on this combined task of school improvement and dropout 
prevention, creating Diplomas Now. The Diplomas Now whole-school reform model seeks to 
transform secondary schools in high-poverty, urban communities so that fewer students drop 
out and more students graduate high school prepared for college and careers. 

The Diplomas Now partnership works with schools to ensure that students are getting 
the support they need to (1) get to school and to class, (2) arrive there ready to leam, and (3) 
keep up with the lessons being taught. In other words, the pathway to student success in 
Diplomas Now schools is linked to attendance, behavior, and course performance; as discussed 
earlier, these “ABCs” are predictive of whether students graduate or drop out. The Diplomas 
Now model is designed to help schools provide the right services to the right students on time 
and at the level of intensity necessary for students to have positive ABC outcomes. To do so, 
Diplomas Now has created a three- tier intervention structure to offer varying levels of support 
for students with different needs: whole-school organizational restructuring and instructional 
reform to strengthen the educational experience of all students (Tier I), individual student 
support (Tier II), and student case management (Tier III). To determine which students need 
extra Tier II or Tier III support the model relies on regular monitoring of Early Warning 
Indicator data, which include measures of students’ attendance, behavior, and course perfor- 
mance. See Box ES. 1 for more on what each of the Diplomas Now partners contributes to the 
overall model. 


The National i3 Evaluation of Diplomas Now: Investigating 
Diplomas Now Implementation and Effectiveness 

Johns Hopkins University, home to Talent Development, was awarded a federal Investing in 
Innovation (i3) validation grant in 2010 to support the expansion and evaluation of Diplomas 
Now. This five-year grant supports the expansion of Diplomas Now from a few schools to more 
than 30 schools across more than 10 school districts. The grant funds also support a rigorous 
experimental evaluation of the Diplomas Now model, the results of which, if positive, will 
“validate” Diplomas Now as an effective secondary school reform model. The evaluation, being 
conducted by two evaluation research firms, MDRC and ICE International, explores not only the 
impact of Diplomas Now but also its implementation, providing lessons to the field about what it 
takes to implement the model and how it rolls out in different school and district contexts. 

Diplomas Now seeks to strengthen secondary schools so that students follow a path that 
leads to their graduation from high school ready for college and careers. The Diplomas Now 
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Box ES.1 

The Diplomas Now Partners and Their Roles 
in a Tiered Intervention System 

Talent Development Secondary 

Talent Development Secondaiy, based at Johns Hopkins University, provides organizational, 
instructional, and curricular support to schools. These Tier 1 interventions are focused on helping 
all students achieve at high levels and preventing them from falling off track. This school-wide 
effort includes reorganizing students and teachers into small learning communities, providing 
professional development and coaching to strengthen teacher pedagogy, and supplying college and 
career preparatory course content. Talent Development Secondary employs a school transfor- 
mation facilitator who works with school leaders to develop a systematic school organization plan 
and oversees the implementation of instructional and curricular reforms. 

For many students, the Tier 1, whole-school organizational and instructional reforms of Talent 
Development Secondary are enough to keep them on track. However, for some students, additional 
and more intensive services are necessary. City Year and Communities in Schools play leading 
roles in providing additional services to these at-risk students. 

City Year 

City Year is an AmeriCorps program through which young adults, ages 18 to 24, participate in a 
year of full-time national service. A team of 10 or more City Year coips members is assigned to a 
school, increasing the number of adults in a building paying attention to students and working with 
them both in and outside of classrooms. The team is led by a City Year program manager and team 
leaders (typically second-year coips members), and the corps members are trained to provide a 
variety of Tier 11 academic and behavioral interventions intended to help students get on track and 
stay on track to graduate. These “near peers” (given their proximity in age to the students) serve as 
tutors, mentors, and role models, personalizing the school experience of the students. In addition, 
the corps members provide after-school programs and help teachers by working with students 
during class time. 

Communities In Schools 

Through a school-based site coordinator, Communities In Schools, a national dropout-prevention 
organization, draws on school and community resources to organize services intended to move the 
students at highest risk of dropping out back on track to graduation. The site coordinator assesses 
the needs of a student, develops an individual case plan for that student that lays out a strategy to 
address those needs, and connects the student to services aligned with the case plan. The provision 
of these services represents Tier III intervention. Examples include professional counseling on 
anger management for a student with severe behavioral issues or long-term tutoring with a subject- 
area expert for a student falling far behind in class. A site coordinator will also provide direct 
service — for example, facilitating small student discussion groups on topics like conflict resolu- 
tion or the transition to adulthood. 


ES-3 




model does this through structural reforms, such as the creation of smaller learning communities 
with teacher teams that work with shared groups of students, and instructional reforms such as 
peer coaching and curricula designed to help students shore up key academic skills. However, 
these whole-school efforts that reach all students are not enough on their own. As discussed 
earlier, the Diplomas Now model also seeks to provide more intensive and targeted support to 
address the needs of students who exhibit the primary ABC indicators of falling off track that 
are predictive of dropping out (poor attendance, negative school behaviors, and course failure), 
particularly during transition years when students are most vulnerable to getting off track: sixth 
grade in middle schools and ninth grade in high schools. Because the timetable of the current 
evaluation is not long enough to follow students through high school graduation, the primary 
research question focuses on measuring the impact of Diplomas Now on ABC outcomes. For 
students in sixth grade and ninth grade, what is the impact of Diplomas Now on three primary 
student outcomes: attendance rates (proportion of enrolled days in attendance), suspensions (in- 
school or out-of-school) and expulsions, and successful course completion? Essentially, does 
the implementation of Diplomas Now result in more students being on track to high school 
graduation by the end of their middle school or high school transition years? 

The evaluation also examines the implementation of the Diplomas Now model, docu- 
menting how this complex, multicomponent reform intended to transform secondary schools is 
implemented by multiple partners. The implementation research explores what it takes to 
implement the model, what factors facilitate or hinder implementation, and the nature of the 
collaboration among multiple actors from the Diplomas Now organizations and the schools. 

In total, 62 schools (33 middle schools and 29 high schools) from 1 1 large urban school 
districts across the country were recruited to participate in the study starting in either the 2011- 
2012 or 2012-2013 school year. Five of the districts are among the 20 largest school districts in 
the country, and all but 1 are among the 100 largest. 6 The participating schools, all eligible for 
Title I funds, serve large populations of low-income and minority students (80 percent eligible 
for free or reduced-price lunches; 83 percent black and Hispanic). Furthermore, the high schools 
participating in the study also have weak promoting power (56 percent), suggesting that they 
struggle to move students from ninth through twelfth grade on time. 7 

Thirty-two of the participating secondary schools were randomly assigned to implement 
the Diplomas Now model (DN schools) and 30 were assigned to continue with “business as 
usual” practices and programs (non-DN schools), either maintaining existing practices and 

6 Chris Plotts and Jennifer Sable, Characteristics of the 100 Largest Public Elementary’ and Secondary’ 
School Districts in the United States: 2007-08, NCES 2010-349, U.S. Department of Education, National 
Center for Education Statistics (Washington, DC: U.S. Government Printing Office, 2010). 

^‘Promoting power” is calculated as the ratio of twelfth-grade students to ninth-grade students three years 
earlier (for example, the ratio of twelfth-graders in 2010-2011 to ninth-graders in 2007-2008). 
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structures within their schools or pursuing other types of school reform. This random assign- 
ment design, often referred to as the “gold standard” in evaluation, creates circumstances under 
which any differences between the two groups of schools (DN and non-DN) that emerge after 
random assignment can be attributed to the program; in short, Diplomas Now caused the 
observed differences. Comparisons of the sizes, staff compositions, and student populations of 
the two groups of schools, as well as the types of programs the schools offered before the start 
of the evaluation, indicate that the two groups of schools were similar to each other before the 
evaluation began. This suggests that random assignment was successful and the non-DN 
schools provide a convincing representation of what would have happened in the DN schools 
had they not implemented the intervention. 

Although the evaluation research will analyze both the impact and the implementation 
of the Diplomas Now model, this report focuses on program start-up and early implementation 
in the DN schools. The report presents the model, describes the schools and school districts that 
are participating in the evaluation, shares findings about first-year implementation fidelity in the 
DN schools, and discusses how program and school staff members collaborated to implement 
the DN model. 


The Diplomas Now Model 

The Diplomas Now model is a multidimensional system of organizational and instructional 
refomis and targeted forms of student support. These elements are classified as nine “inputs” 
that the Diplomas Now partners implement in collaboration with school personnel. Some of 
these inputs represent substantial interventions on their own, such as implementing a rigorous 
curriculum or setting up a tiered system to identify at-risk students and tailor responses to their 
specific needs. Diplomas Now integrates these component interventions into a cohesive model 
focused on ensuring that all students have a path to graduation. Eight of these inputs align with 
the Four Pillars of Diplomas Now, a characterization of the model used by Diplomas Now staff 
members to help them organize their work. The Four Pillars and their associated inputs are 
presented in Figure ES. 1 . 

Pillar I: Teacher Teams and Smaller Learning Communities 

Diplomas Now collaborates with school leaders to organize schools such that small 
groups of teachers work with the same population of students. These smaller learning communi- 
ties create opportunities for personalization, where teams of teachers know the same students 
and can work together to best teach and support them. Students also see the same peers in their 
classes and become known to one another. These teacher teams and smaller learning communi- 
ties function best when there are opportunities for teachers to collaborate within the daily 
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Diplomas Now 
Figure ES.l 
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schedule and when they have classes long enough to cover material in depth and keep up the 
pace of instruction. 

Pillar II: Curriculum and Instruction with Professional Development 

This pillar is focused on teaching and learning, and on giving teachers the training and 
resources they need to deliver strong lessons. Through professional development activities and 
instructional coaching, teachers have an opportunity to sharpen their pedagogy. Having curricu- 
lar materials aligned with college- and career-ready standards means that teachers have useful 
content to deliver to students. Professional development and curricular materials for accelerated 
remediation courses for struggling students are also aspects of this pillar. 

Pillar III: Tiered Student Supports 

Providing more intensive support for students with greater needs is the core idea of this 
pillar. The tiered intervention model involves implementing an early warning system that draws 
on data on the ABC indicators of individual students. It relies on staff members who work with 
a group of students in common having regular times to meet to review those data and to plan 
interventions for students who are off track or at risk of going off track. 

Pillar IV: Can-Do Culture and Climate 

School reform is difficult, and school staff members often have too much to do when 
they are asked to make change. Diplomas Now brings over a dozen staff members to a school to 
help coordinate school transformation, provide support to the school’s staff, provide additional 
services to students, and engage with families and community organizations. Providing and 
organizing resources to assist the school’s staff helps foster a culture and climate where it feels 
possible to improve the school and support students better. 

The Four Pillars and their eight associated inputs are supported by a ninth input, Pro- 
gram Staff Training and Development, which involves providing the skills and knowledge to 
Diplomas Now staff members that they need to implement the other eight. This includes 
training during the summer and during the school year for school-level Talent Development 
Secondary, City Year and Communities In Schools staff members. To ensure successful 
implementation at each school, each of the Diplomas Now partner organizations has a support 
system for school-based staff members that includes local and national program experts with 
strong relationships to school districts . 8 


Additional information about Diplomas Now can be found on the Internet at http://diplomasnow.org. The 
Diplomas Now website includes information about the partner organizations and the model, as well as contact 
information for the organization. 
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Implementation of the nine inputs is hypothesized to affect a series of outcomes. (See 
Figure ES.l.) Initially, model implementation is supposed to lead to early-stage changes in 
school outcomes such as the quality of school climate and communication among stakeholders, 
and in student outcomes like study habits and attitudes about and engagement with school. 
These changes, in turn, are expected to lead to impacts on intermediate outcomes — the ABC 
indicators of attendance, behavior, and course performance. Positive impacts on ABC outcomes 
should then lead to increased high school graduation rates. 


Fidelity of Implementation of Diplomas Now in the First Year 

The evaluation of Diplomas Now begins with an assessment of the fidelity with which the 
Diplomas Now model was implemented in the 32 schools randomly assigned to do so. That is, 
in the first year of implementation (2011-2012 for first- wave schools and 2012-2013 for 
second-wave schools), how well did the model as implemented match its design? How much of 
the Diplomas Now school-improvement effort was put into place in these 17 middle schools 
and 15 high schools? 

Several primary findings regarding fidelity in the first year emerged from the imple- 
mentation analysis. 

• After the first year, overall model implementation in the 32 DN schools has 
gotten underway and achieved some traction. On average DN schools suc- 
cessfully implemented 61 percent of 111 separate program components 
across the nine inputs. 

Complex, multifaceted whole-school reforms like Diplomas Now typically take a few 
years to reach full implementation. Getting the majority of model components implemented in 
the first year suggests that schools are off to a promising start with this reform. 

• However, none of the DN schools were able to implement all of the 62 com- 
ponents of the model believed to be most critical by the Diplomas Now or- 
ganizations. 

The Diplomas Now Implementation Support Team identified a subset of the program 
components that it hypothesized were the most likely to lead to the desired changes in school 
and student outcomes. During the first year of implementation, the DN schools struggled to 
implement some of these “critical” components. For example, in some schools, school-level 
Diplomas Now staff members were not in place to begin work with school persomiel by the 
start of the academic year. Many middle schools did not get the Diplomas Now reform 
curricula in place. Only about half of the schools managed to hold Early Warning Indicator 
meetings weekly. And although instructional support for English/language arts and math 
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teachers was provided at many schools, instructional coaches struggled to work with teachers 
as often as the model design prescribed. So while the schools got off to a generally strong start 
with implementation, it may be that they could have focused more intently on some aspects of 
the model hypothesized to be the most important. 

• DN schools were most successful during the first year in offering schools in- 
tegrated on-site support, implementing a tiered intervention model, and add- 
ing student support services like tutoring and after-school programs into the 
existing school structure. 

Taken together, this means that Diplomas Now was most successful at combining data- 
driven identification of student needs (catching students who are off track) with the means to 
respond to those needs through coordinated programs and personnel (getting those students 
back on track). These are essential functions of the Diplomas Now focus on students’ progress 
toward high school graduation. 

• DN schools were less successful in adopting new curricula and implementing 
peer coaching models, both of which require gaining the trust and investment 
of school administrators and teachers, and which may require additional time 
to implement as a result. 

Qualitative data reveal that school staff members were not always convinced of the val- 
ue of implementing new curricula, and that some teachers were reluctant to be “coached.” 
These findings speak to the importance of giving a school’s staff a role in selecting some 
school-level program staff members, if possible, to encourage greater rapport and trust between 
school and Diplomas Now staff members. This is especially the case for instructional coaches, 
who frequently interact with teachers on a one-on-one, peer-to-peer basis. 


Collaborative Interactions Among Diplomas Now 
Partners and School Staff Members 

School improvement necessitates that people act, react, and interact in new ways, and effective 
collaboration is essential to such change. Effective collaboration is at the heart of the complex 
Diplomas Now school reform, which deploys staff members from Talent Development, City 
Year, and Communities In Schools to work in concert with each other and with school staff 
members to implement the tiered model in school buildings. As designed, the program needs 
individuals from the three partner organizations to establish shared norms for working together, 
communicating continuously, sharing leadership responsibilities, and making decisions together 
in order to achieve the program goals of addressing student attendance, behavior, and course 
performance. 
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As part of its research, the evaluation team investigated the collaborations that were part 
of first-year implementation. These collaborations did not only include those among staff 
members from the three Diplomas Now partner organizations, but also those between Diplomas 
Now program staff members and school personnel. 

• Two aspects of collaboration appeared to be most important across schools 
and various role groups: investment and role clarity. 

Administrators and teachers are key stakeholders whose engagement in implementing 
the Diplomas Now model inputs is indispensable. In order to become actively engaged, it is 
important that they understand the model through activities such as information sessions and 
meeting school staff members at other DN schools. Continual communication, including regular 
meetings and informal check-ins, helps build the trust and acceptance necessary for the collabo- 
rative work of model implementation. 

In addition, it is crucial that Diplomas Now school-based staff members establish their 
purposes and roles, with teachers and administrators as well as among themselves. The influx 
of Talent Development, City Year, and Communities In Schools staff members provided 
schools with the human resources necessary for the implementation of the Diplomas Now 
model. However, increased clarity regarding the roles and responsibilities of these Diplomas 
Now staff members would have further improved model implementation in the first year. 
Considering the numerous program components implemented at DN schools, and the number 
of additional staff members enlisted to support implementation, it is understandable that both 
Diplomas Now program staff members and school staff members experienced uncertainty 
about each other’s roles and expectations. Especially during the first year, it would be very 
helpful to clarify these roles, responsibilities, and expectations through additional guidance and 
program staff training, school staff training, and joint training. In their continuing work in the 
schools in this study, Diplomas Now has been taking such steps. Furthermore, hiring Diplomas 
Now school-based staff members before the academic year begins appears to facilitate cooper- 
ation that results in quicker implementation of short-term program goals and better planning 
for longer-term initiatives. 


Next Steps 

This report is the first of three planned for this evaluation. Over the course of the evaluation, the 
impact and implementation research presented in these three reports will tell the story of what 
Diplomas Now is, what it took to implement the Diplomas Now model, what it looked like as 
implemented in schools around the country, and how it ultimately differed from business as 
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usual in the districts in which it was implemented. In addition, the evaluation will provide 
evidence about the impact of Diplomas Now on schools and students. The next two reports will 
present findings about the continuing implementation of Diplomas Now, whether implementa- 
tion of the model is changing DN schools and making them different in their organization and 
practices from non-DN schools, and finally the impact of Diplomas Now on student outcomes. 
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Chapter 1 

Introduction 


High school graduation rates in the United States have been on the rise. Mostly stagnant from 
1970 to 2000, the national graduation rate has risen steadily from 2000 to 2010. 1 The graduation 
rate rose from 71.7 percent in 2001 to 78.2 percent in 2010, and the number of high schools 
where large proportions of students are unlikely to graduate declined from 2,007 in 2002 to 
1,424 in 201 1. 2 While this is good news for the nation, the reality is that too many students (one 
in five) still do not complete high school within four years. Even refonns that have significantly 
improved high school graduation rates have not solved the dropout problem. For example, 
district- wide reform in New York City has increased the number of nonselective small high 
schools there, implemented a choice system that gives students the opportunity to rank the high 
schools they would like to attend, and closed large, failing high schools; this reform has resulted 
in positive impacts on four-year high school graduation rates that are close to 10 percentage 
points. However, the students benefiting from attending the newer small schools still only 
graduate at a 70 percent rate, meaning that plenty do not finish on time, if at all. 3 Also, even if 
graduation rates continue to rise in school districts across the country, the decreasing number of 
students who do not graduate will represent more and more those students who face the most 
serious barriers to earning their diplomas, those who most need intensive academic, social, and 
other interventions to make it through high school. 

Researchers know more and more about the factors that predict which students are like- 
ly to drop out and not graduate from high school. Earning course credits and not failing courses 
in ninth grade is highly predictive of being on track to graduate high school. 4 The ninth-grade 
year is a particularly important time of transition for students, and represents the time when 
students are most likely to drop out of school. 5 Signs of students being at risk of struggling in 
ninth grade and eventually dropping out are also evident in middle school. As early as the sixth 
grade, 60 percent of future school dropouts in high-poverty schools exhibit indicators of falling 
off track — poor attendance, poor behavior, and poor course perfomiance (that is, course 
failure). 6 These complementary on- track and off-track indicators suggest ways to intervene with 


'Mumane (2013). 

‘Balfanz, Bridgeland, Bruce, and Fox (2013) refer to these schools as “dropout factories,” defined as high 
schools where twelfth-grade enrollment is 60 percent or less of the ninth-grade enrollment three years earlier. 
3 Bloom and Unterman (2013). 

4 Allensworth and Easton (2005). 

5 Herlihy and Kemple (2004); Quint (2006). 

6 Balfanz, Herzog, and Mac Iver (2007). 
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students in secondary schools that could increase their likelihood of graduating. Given how 
pivotal the ninth grade is, high school interventions could prepare to respond promptly to signs 
of students going off track during that crucial transition year and include support for on-track 
behaviors and performance. Given that students may start to exhibit off-track behaviors in the 
middle grades, middle school interventions could focus on addressing those behaviors early, as 
students transition into middle school, and then on preparing them for a successful transition 
into high school. 

We also know where to find many of the students most at risk of dropping out. Approx- 
imately half of all high school dropouts attend high schools where the size of the twelfth-grade 
class is 60 percent or less of its ninth-grade size three years earlier. These high schools, and the 
middle schools that feed into them, are located predominantly in low-income, urban areas. 7 
Given the concentration of students at risk of dropping out in these communities and schools, 
intensive reform efforts are probably necessary to help them overcome challenges inside and 
outside of school that impede their paths to graduation. Also, given the concentration of the 
dropout problem, reform efforts that involve structures and practices explicitly intended to 
address factors predictive of dropping out could help reduce the number of dropouts and help 
graduation rates continue to rise. 

In the 21st century, supporting students through high school graduation is not usually 
enough to guarantee that they have the tools to live self-sufficient and productive lives. Most 
students also aspire to attend college, and need to leave high school with the skills and 
knowledge required to succeed in postsecondary education. Unfortunately, many students are 
not leaving high schools with these skills even when they succeed in graduating. Across the 
nation, roughly 40 percent of college undergraduates enroll in at least one developmental 
education course (remedial, non-credit-bearing courses that students are required to take if they 
do not meet the knowledge and skill standards of credit-bearing courses). That number is far 
greater for students in community college (where many low-income students from urban areas 
enroll), where almost 60 percent of students enroll in at least one developmental course. 8 
Beyond supporting struggling students to graduation, school improvement efforts need to 
ensure that all students participate and succeed in rigorous curricula that prepare them for 
college and careers. 


7 Balfanz, Bridgeland, Bruce, and Fox (2013). 

s Attewell, Lavin, Domina, and Levey (2006). 


2 



Diplomas Now: Transforming Schools and Responding 
to the Dropout Challenge 

Three national organizations — Talent Development Secondary, City Year, and Communities 
In Schools — have partnered to take on this combined task of school improvement and dropout 
prevention, creating Diplomas Now. The Diplomas Now whole-school reform model seeks to 
transform secondary schools in high-poverty, urban communities so that fewer students drop 
out and more students graduate high school prepared for college and careers. 

The Diplomas Now partnership works with schools to ensure that students are getting 
the support they need to (1) get to school and to class, (2) arrive there ready to learn, and (3) keep 
up with the lessons being taught. In other words, the pathway to student success in Diplomas 
Now schools is linked to attendance, behavior, and course perfonnance — the “ABCs” predic- 
tive of whether students graduate or drop out. The Diplomas Now model is designed to help 
schools provide the right services to the right students on time and at the level of intensity 
necessary for students to have positive ABC outcomes. To do so, Diplomas Now has created a 
three-tier intervention structure that offers varying levels of support for students with different 
needs. The Diplomas Now model relies on regular monitoring of Early Warning Indicator 
(EWI) data, which include measures of students’ attendance, behavior, and course perfonnance, 
to determine which students need extra support. 

The first tier is focused on helping all students achieve at high levels and preventing 
them from falling offtrack. Talent Development Secondary, based at Johns Hopkins University, 
provides organizational support to schools, helping leaders structure their schools to produce 
conditions under which students are more likely to succeed. Specifically, it helps schools 
reorganize students and teachers into small learning communities, within each of which an 
interdisciplinary group of teachers can collaborate and focus their attention on a smaller group 
of shared students. Talent Development Secondary also provides instructional and curricular 
support to schools, seeking to strengthen pedagogy and course content so that teachers can 
better educate students. To do all this, Talent Development Secondary employs a school 
transformation facilitator who works with school leaders to develop a systematic plan integrat- 
ing the host of services the Diplomas Now team can bring to a school. The school transfor- 
mation facilitator also oversees curricular reforms at the school and works with Eng- 
lish/language arts and math instructional coaches to ensure the school is employing college 
readiness standards for all students, helping teachers improve their pedagogy, and offering 
ample remediation for students struggling to meet college readiness goals. 

For many students, the Tier I, whole-school organizational and instructional reforms of 
Talent Development Secondary are enough to keep them on track. However, for some students, 
additional and more intensive services are necessary. Through regular review of the ABC 
indicators, Diplomas Now and school staff members identify students who are starting to get off 
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track based on slipping attendance, more frequent misbehavior, or declining performance in 
class. The school transformation facilitator coordinates meetings to discuss students’ EWI data 
with Diplomas Now school-based staff members, interdisciplinary teams of teachers, and other 
support personnel at the school (counselors, for example). During these meetings, staff mem- 
bers discuss possible causes for individual students’ off-track behaviors and determine what 
actions to take to help those students get back on track to graduation. Much of the work of City 
Year and Communities in Schools is in supporting these at-risk students. 

City Year is an AmeriCorps program through which young adults, ages 18 to 24, partic- 
ipate in a year of full-time national service. City Year corps members, led by a program manag- 
er and supported by experienced (second-year) corps members who act as team leaders, are 
trained to provide individual, small-group, and school-wide academic and behavioral interven- 
tions intended to help students get on track and stay on track to graduate. As part of the Diplo- 
mas Now model, a team of 10 or more City Year corps members is assigned to a school, 
increasing the number of adults in a building paying attention to students and working with 
them both in and outside of classrooms. These “near peers” (given their proximity in age to the 
students) serve as tutors, mentors, and role models, personalizing the students’ school experi- 
ence through one-on-one relationships. The corps members also provide after-school programs. 
These services strengthen Tier I whole-school reforms and are considered a critical part of the 
Tier II interventions meant for students who are showing signs of falling off track on one or 
more of the ABC indicators. 

Communities In Schools is a national dropout-prevention organization that has been 
working with schools for over 30 years. Through a school-based site coordinator, Communities 
In Schools draws on in-school and community resources to organize both school-wide and 
individual services intended to address obstacles to students’ progress to graduation. Within the 
Diplomas Now model, the Communities In Schools’ site coordinator is a critical player in 
addressing the needs of the students facing the most challenges and at the highest risk of 
dropping out, as measured by the ABC indicators. The site coordinator supports these students 
through a set of Tier III interventions. The site coordinator assesses the needs of a student, 
develops an individual case plan for that student that lays out a strategy for addressing those 
needs, and then connects the student to services in and outside of school that align with the case 
plan. Examples of these Tier III intervention services include professional anger-management 
counseling for students with severe behavioral issues, and long-term tutoring with subject-area 
experts for students falling particularly far behind in those subjects. Not only might a site 
coordinator connect a student to other service providers, but he or she typically provides direct 
service as well — for example, facilitating small student discussion groups on topics specific to 
needs they might have, such as conflict resolution or the transition to adulthood. 
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The team of Diplomas Now and school staff members continues to use EWI data to 
monitor the progress of all struggling students receiving Tier II or Tier III interventions intended 
to help them get back on track. The level of service a student receives may change depending 
on how long that student is off track and on whether his or her ABC indicators grow more 
negative or positive over time. Talent Development Secondary’s organizational and instruction- 
al refonns are thus bolstered by the programs and staffs of City Year and Communities In 
Schools, which address the academic, behavioral, and social challenges faced by individual 
students, particularly those students most at risk of dropping out. All three groups also work to 
strengthen parent and community involvement in the school by sponsoring family and commu- 
nity events, working with community leaders to identify new resources for the school and 
students, and involving parents more directly in students’ education and progress. 

Ideally, the Diplomas Now team starts working with a school during the spring prior to 
the first year of implementation. At that time, Diplomas Now can work with the school admin- 
istrators to plan program implementation and identify and hire the new staff. This also allows 
school administrators and staff members to attend the Diplomas Now summer institute before 
the first implementation year, and allows ample time for the training of Diplomas Now staff 
members at the school. The Diplomas Now staff focuses its attention on students in transition 
years (sixth and ninth grades) during the first year of implementation. Support in these transition 
years is the strongest because they are so important in a student’s trajectory toward graduation. 
In subsequent years, the Diplomas Now staff follows and keeps supporting these students, but 
also continues to add resources in order to support and follow new cohorts of sixth- and ninth- 
grade students. 

Prior research on the Talent Development Secondary whole-school reform model has 
shown evidence of improvement in the academic achievement, school progress, and attendance 
of students in urban secondary schools . 9 However, on their own Talent Development Second- 
ary’s organizational and instructional refonns cannot address the multiple challenges that some 
students from low-income communities face that put them on a pathway to dropping out. 
Although Kernple, Herlihy, and Smith found that Talent Development Secondary had positive 
impacts on students’ academic achievement and ninth-grade completion, they also indicated 
that many students were still not making adequate progress. They suggested that programs like 
Talent Development Secondary needed more “power” in order to have success with more 
students. They also indicated that Talent Development Secondary was not having an impact on 
students who repeated the ninth grade, one example of the kinds of students who need more 
intensive intervention if they are to make it to graduation . 10 By collaborating with City Year and 

9 Kemple, Herlihy, and Smith (2005). 

10 Kemple, Herlihy, and Smith (2005). 
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Communities In Schools in Diplomas Now, Talent Development Secondary is now part of a 
more powerful refonn model that combines its whole-school refonns with City Year and 
Communities In Schools staff members and programs that focus on the students most at risk of 
dropping out. The three organizations’ unique partnership represents a multidimensional 
approach to school refonn that includes structural and cunicular refonns for the entire school, 
professional development opportunities for teachers and school staff members, a data-driven 
student tracking system to identify at-risk students, and an auxiliary staff to support teachers and 
other school staff members and to work directly with at-risk students. 

These various organizational and instructional refonns and targeted student support ser- 
vices are displayed as Diplomas Now model “inputs” in Figure 1.1. These nine inputs represent 
elements of the model that the Diplomas Now partners implement in schools with the collabora- 
tion of the school’s staff As noted earlier, Diplomas Now is a multidimensional system of 
refomis, and some of these inputs represent substantial interventions on their own — for 
example, implementing a rigorous curriculum or setting up a tiered system to identify at-risk 
students and tailor responses to their specific needs. Diplomas Now integrates these component 
interventions into a cohesive model focused on ensuring that every student has a path to 
graduation. Eight of these inputs align with the Four Pillars of Diplomas Now, a characteriza- 
tion of the model used by Diplomas Now staff members to help them organize their work: 

I. Teacher Teams and Smaller Learning Communities 

II. Curriculum and Instruction with Professional Development 

III. Tiered Student Supports 

IV. Can-Do Culture and Climate 

Pillar I is focused on school organization. Pillar II is about instructional refonn, support- 
ing stronger teaching and providing meaningful cunicula. Pillar III is about aligned student 
support — matching the intensity of service or support to the level of need a student exhibits. 
Pillar IV is focused on (1) ensuring that the Diplomas Now staff members requisite for model 
implementation are in place at a school and (2) building positive attitudes about school change. 
The pillars and their associated inputs are described in more detail below. 

Pillar I: Teacher Teams and Smaller Learning Communities. As discussed earlier, Di- 
plomas Now collaborates with school leaders to organize schools such that small groups of 
teachers work with the same students. These smaller learning communities create opportunities 
for personalization, where teams of teachers know the same students and can work together to 
best teach and support them. Students also see the same peers in their classes, and become 
known to one another. These teacher teams and smaller learning communities function best 
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Diplomas Now 
Figure 1.1 
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when there are opportunities for teachers to collaborate within the daily schedule, for example 
through common planning time, and when they have class periods long enough to cover 
material in depth and keep up the pace of instruction. The input “Strong Learning Environ- 
ments” captures these organizational aspects of this pillar. 

Pillar II: Curriculum and Instruction with Professional Development. This pillar is fo- 
cused on teaching and learning, and on giving teachers the training and resources they need to 
deliver strong lessons. Two inputs are aligned with this pillar — “Professional Development 
and Peer Coaching” and “Curriculum for College Readiness.” Through professional develop- 
ment activities and instructional coaching, teachers have an opportunity to sharpen their 
pedagogy. Having curricular materials aligned with college- and career-ready standards means 
that these teachers have useful content to deliver to students. These two inputs also cover 
professional development, along with curricular materials for accelerated remediation courses 
for struggling students. 

Pillar III: Tiered Student Supports. Providing more intensive support for students with 
greater needs is the core idea of this pillar, and the three inputs aligned with this goal are 
“Tiered Intervention Model,” “Student Supports,” and “Student Case Management.” The tiered 
intervention model involves implementing an early warning system, which in turn requires 
access to data on the ABC indicators of individual students. It also means that the staff members 
who work with a group of students in a smaller learning community need regular times to meet 
to review those data and to plan interventions for students who are off track or at risk of going 
off track. City Year and Communities In Schools must then be able respond to off-track 
students’ needs, supplementing the efforts of classroom teachers through Tier II and Tier HI 
interventions. 

Pillar IV: Can-Do Culture and Climate. School reform is difficult, and the school staff 
often has too much to do when asked to make change. It is important to make change feel 
possible and create a positive environment of improvement, so that the school staff members 
can buy into reforms, develop high expectations, and see the reform effort as a coordinated 
whole and not small, disconnected pieces. The inputs of “Integrated On-Site Support” and 
“Family and Community Involvement” are about providing additional support to schools to 
help them do so. Diplomas Now brings over a dozen staff members to a school to help coordi- 
nate school transfonnation, provide support to the school’s staff, and provide additional services 
to students. Figure 1.2 displays these school-based Diplomas Now staff members. In addition, 
Diplomas Now staff members seek to involve families and communities in their schools to 
provide additional support to students. Providing and organizing resources to assist the school’s 
staff helps foster a culture and climate where it feels possible to improve the school and support 
students better. 



Diplomas Now 
Figure 1.2 
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students 


* 


Diplomas Now Local Executive Teams 

Regional and district-level leaders who support the school-level staff 

Talent Development district field managers, instructional facilitators, and 
school and student support services (S4) facilitators 

Communities In Schools regional executive directors 

City Year managing directors 

* 

Diplomas Now Implementation Support Team (PN1ST) 

National team with representatives from all three organizations that supports local executive teams 
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The ninth input, “Program Staff Training and Development,” shown at the far left of 
Figure 1.1, is about providing the skills and knowledge to Diplomas Now staff members that 
they need to implement the other eight model inputs under the Four Pillars. This includes 
training for school-level Talent Development Secondary, City Year, and Communities In 
Schools staff members during the summer prior to the school year, and continuing training 
throughout the school year. To ensure successful implementation at each school, each of the 
Diplomas Now partner organizations has a support system for school-based staff members that 
includes local and national program experts with strong relationships to school district staffs. 
The Talent Development Secondary school transfonnation facilitator, City Year program 
manager, and Communities In Schools site coordinator work with school administrators to 
oversee the implementation of various program inputs. They also work together to ensure clear 
lines of communication and support between the school’s program staff and the local executive 
teams from each organization, which in turn are in communication with the national-level 
Diplomas Now Implementation Support Team, as shown in Figure 1.2. 11 

Figure 1.1 also shows the hypothesized connection of the model’s inputs to early, in- 
termediate, and longer-term school and student outcomes. Initially, the model is supposed to 
lead to changes in school outcomes such as school climate and communication among stake- 
holders, and in student outcomes such as study habits and attitudes about and engagement with 
school. These changes, in turn, are expected to lead to impacts on intennediate outcomes — the 
ABC indicators of attendance, behavior, and course perfonnance. Positive impacts on ABC 
outcomes should then lead to increased high school graduation rates. Given the complexity of 
the Diplomas Now model, it can take up to three years for a school to fully implement all 
components of the model at a high level. Evaluation research makes it possible to understand 
the implementation of the Diplomas Now model and how it progresses over time, as well as its 
effects on schools and students. The next section of this chapter describes the national evalua- 
tion of the implementation and impact of the Diplomas Now secondary school refonn model. 
What does it take to implement the Diplomas Now model? How does model implementation 
make schools different from others struggling with the same challenges and frying other school 
improvement strategies? Does implementation lead to the hypothesized changes in school and 
student outcomes? 


"Additional information about Diplomas Now can be found on the Internet at http://diplomasnow.org. 
The Diplomas Now website includes information about the partner organizations and the model, as well as 
contact information for the organization. 
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The National i3 Evaluation of Diplomas Now: Investigating 
Diplomas Now’s Implementation and Effectiveness 

Johns Hopkins University, home to Talent Development Secondary, was awarded a federal 
Investing in Innovation (i3) validation grant in 2010 to support the expansion and evaluation of 
Diplomas Now. Box 1.1 describes the i3 program. There were almost 1,700 i3 grant applica- 
tions in 2010, and the i3 validation grant won by Johns Hopkins University that year was one of 
49 awarded, of which 15 were validation grants. This five-year validation grant was intended to 
support the expansion of Diplomas Now from a few schools to more than 30 schools across 10 
or more school districts. At the same time, the grant funds support a rigorous experimental 
evaluation of the Diplomas Now model, the results of which, if positive, will “validate” Diplo- 
mas Now as an effective secondary school reform model. The evaluation is intended not only to 
assess the impact of Diplomas Now but also to explore its implementation, providing lessons to 
the field about what it takes to implement the model and how it rolls out in different school and 
district contexts. 

MDRC and ICF International, policy research finns independent of the Diplomas Now 
partner organizations, are conducting the national i3 evaluation of Diplomas Now. The founda- 
tion of the evaluation is a school-level random assignment design. That is, within school 
districts participating in the i3-funded expansion of Diplomas Now, schools eligible for model 
implementation are assigned randomly (through a lottery-like process) to implement Diplomas 
Now (DN schools) or to continue with business as usual (non-DN schools), either continuing 
existing practices and structures within their schools or pursuing other types of school reform. 
Random assignment is often referred to as the “gold standard” in evaluation design because it 
creates circumstances under which any differences between the two groups of schools (DN and 
non-DN) that emerge after random assigmnent can be attributed to the program (Diplomas 
Now): Diplomas Now caused the observed differences. 

As discussed earlier, the Diplomas Now model seeks to address the needs of students 
who exhibit the primary ABC indicators of falling off track that are predictive of dropping out: 
poor attendance, negative school behaviors, and poor course performance (particularly course 
failure). Students are also more vulnerable to getting off track during transition years, typically 
sixth grade in middle schools and ninth grade in high schools. Therefore, the evaluation 
measures the impact of Diplomas Now on ABC outcomes during students’ sixth- and ninth- 
grade years. For sixth-grade and ninth-grade students, the evaluation asks three primary re- 
search questions: 

1. What is the impact of Diplomas Now on students’ attendance rates (proportion of 
enrolled days in attendance)? 
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Box 1.1 


The Investing in Innovation (i3) Fund 

The i3 fund is a U.S. Department of Education program created as part of the American 
Recovery and Reinvestment Act of 2009 that provides funding via competitive grants to 
school districts, consortia of schools, or nonprofit organizations partnered with school districts. 
It supports the implementation and evaluation of innovative, promising, and proven practices 
intended to address persistent educational challenges. The U.S. Department of Education has 
funded 92 unique i3 projects thus far. Each applicant must provide evidence of effectiveness or 
promise and must propose an evaluation of the practice, strategy, or program as part of the 
grant application so that each project will contribute to the larger body of knowledge about the 
implementation and impact of educational programs in the field. The grants are organized into 
three tiers, which vary in the amount of prior evidence needed, the rigor of the evaluation 
expected to coincide with implementation, and the amount of funding available: 

1) Scale-up grants provide funding to expand to the national level practices, strategies, 
or programs for which there is strong prior evidence of effectiveness. Evaluators are 
expected to provide evidence of effectiveness at a national scale. 

2) Validation grants provide funding to expand at the regional level practices, strate- 
gies, or programs that show promise, but for which there is currently only moderate 
evidence of effectiveness. Evaluators are expected to show evidence of effectiveness. 

3) Development grants support the development of high-potential but relatively untest- 
ed practices, strategies, or programs. Evaluators are expected to show evidence of 
promise. 

SOURCE: U.S. Department of Education (2013). 


2. What is the impact of Diplomas Now on the proportion of enrolled days suspended 
(in school or out of school) or expelled? 

3. What is the impact of Diplomas Now on the proportion of attempted core courses 
passed? 

Essentially, does the implementation of Diplomas Now result in fewer students getting 
off track and more students staying on track to eventual high school graduation during their 
transition years into middle school or high school? The impact analyses to answer these ques- 
tions will focus on sixth- and ninth-grade students in the second year after random assigmnent, 
when DN schools will be in their second year of implementing the model and at least some staff 
members will already have a year of experience working within the Diplomas Now framework. 
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The focus is on second-year impacts because research suggests that it takes time for the true 
impacts of whole-school reform, representing large-scale change in a school, to emerge. 12 

The evaluation will also answer related, exploratory research questions. Although the 
timeline of the evaluation is not long enough to follow high school students through graduation 
or middle school students through their ninth-grade transition year into high school, the study 
team will analyze the impact of Diplomas Now on the ABC outcomes of seventh- and tenth- 
graders. In addition, the research will investigate impacts on other attendance, behavior, and 
course perfonnance outcomes, most notably the impact of Diplomas Now on a composite 
student “on-track” indicator based on achieving 90 percent attendance, not being suspended, 
passing math and English/language arts courses, and being promoted on time to the next grade. 
The study team will also analyze the impact of Diplomas Now on outcomes believed to be 
precursors to changes in ABC outcomes: the early outcomes presented in Figure 1.1, such as 
changes in school climate, the availability of after-school programs, and students’ self- 
confidence, effort in school, and study habits. Furthermore, the evaluation will explore whether 
the impacts vary based on school or student characteristics, such as school level (middle or 
high) or whether students were already on or off track when model implementation started. 

To complement the research on the impacts of Diplomas Now, the evaluation also in- 
cludes research on the implementation of the model. The Diplomas Now model is a complex, 
multicomponent reform that brings together multiple partners. Documenting the implementa- 
tion of this complex reform in districts and schools around the country over the next few years 
will inform the field about implementing partnership models like this one. It will also add 
context to the impact evaluation and allow the research team to explore the relationship between 
how faithfully the model is implemented and student outcomes. The implementation research 
will investigate what it takes to implement the Diplomas Now model, and whether the national 
implementation of Diplomas Now in DN schools resulted in differences in organization and 
practice from the non-DN schools. The evaluation plans to address the following research 
questions: 

1. How did the intervention as implemented compare with the intervention as 
planned? 

2. How much variation in implementation fidelity was there across sites? 

3. What were the factors that facilitated or hindered implementation of the Diplomas 
Now model? 


12 Borman, Hewes, Overman, and Brown (2003). 
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4. How did DN schools compare with non-DN schools in their structures, programs, 
and practices? In other words, did the implementation of Diplomas Now create a 
difference in the structures, programs, and practices of DN schools and non-DN 
schools? 

In combination, the impact and implementation research will provide valuable infor- 
mation about Diplomas Now. The evaluation will tell the story of what Diplomas Now is, what 
it took to implement the Diplomas Now model, what it looked like as implemented in schools 
around the country, and how ultimately it differed from business as usual in those schools’ 
districts. In addition, the evaluation will provide evidence about how Diplomas Now affects 
schools and students, and about whether it makes a difference. 


Orientation to This Report 

This report is the first of three planned as part of this evaluation. These three reports are orga- 
nized to follow the narrative of Diplomas Now implementation. This first report focuses on 
program start-up and early implementation in the DN schools. The second report will present 
findings about the implementation of Diplomas Now in the second year, and whether imple- 
mentation of the model is making DN schools different in their organization and practices from 
non-DN schools. In addition, it will discuss possible early-stage changes that Diplomas Now 
might produce in schools and students, presenting the impacts of Diplomas Now on early 
outcomes such as the availability of after-school programs and student engagement with and 
effort in school. The third and final report will focus on answering the primary research ques- 
tions of this evaluation, presenting Diplomas Now’s impacts on student attendance, behavior, 
and course perfomiance. 

The rest of this report will present information about the schools participating in the 
evaluation, share findings about first-year implementation fidelity in the DN schools, and 
discuss how different groups collaborated in the implementation of the DN model. Chapter 2 
presents information on the study sample and discusses the assigmnent of schools at random to 
implement Diplomas Now or to continue with business as usual. The chapter describes the 
schools and districts participating in the study and how they were recruited, and compares the 
randomly assigned DN and non-DN schools at baseline (that is, at the start of the study, before 
Diplomas Now model implementation began in the DN schools). In addition, the chapter 
discusses whether DN and non-DN schools had preexisting characteristics similar to features or 
practices that are part of the Diplomas Now model. Chapter 3 presents findings about the 
implementation of the Four Pillars of Diplomas Now and the associated program inputs, 
including how much of the model was implemented in DN schools in the first year. The fourth 
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and final chapter describes the collaboration of staff members from the three Diplomas Now 
organizations with each other and with school administrators and teachers, and considers factors 
that facilitated or hindered collaboration. 
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Chapter 2 


School Districts and Schools Participating 
in the Diplomas Now i3 Evaluation 


The national evaluation of Diplomas Now has, as its foundation, a study sample of 62 second- 
ary schools from 1 1 urban school districts around the country. These schools were assigned 
through a lottery-like process (that is, random assigmnent) either to implement the Diplomas 
Now model or to continue with “business-as-usual” practices and programs already in place or 
planned at their schools. This chapter discusses the recruitment of these schools and districts by 
the Diplomas Now partner organizations, explains their assignment to implement the Diplomas 
Now model or not, and describes how the schools compare with others in their districts and the 
nation. The chapter also discusses how the two assigned groups of schools compare with one 
another, and provides preliminary information about whether practices and programs similar to 
those that are part of the Diplomas Now model were in place at these schools prior to the start of 
Diplomas Now model implementation. 

This chapter will make several main points about the study sample: 

• The 62 schools (33 middle schools and 29 high schools) recruited to partici- 
pate in the evaluation fit the characteristics of schools for which the Diplo- 
mas Now model is designed. The 1 1 participating school districts tend to be 
large and midsize urban districts, and the participating schools have high 
concentrations of high-needs students. Since the Diplomas Now model is de- 
signed to assist schools struggling with high dropout rates, it is important to 
note that, prior to the evaluation, the high schools participating in the study 
had a lower average rate of success moving students from ninth through 
twelfth grades than the high schools in the participating school districts as a 
whole, and than high schools across the country as a whole. 

• Overall, the 32 schools implementing the Diplomas Now program (DN 
schools) and the 30 comparison schools not implementing the program (non- 
DN schools) were not statistically different before the start of the evaluation 
in various measures of student body size, staff size, racial or gender composi- 
tion, setting, or student socioeconomic status. This similarity suggests that 
the random assigmnent of schools was successful and that the non-DN 
schools provide a convincing representation of what would have happened in 
the DN schools had they not implemented the intervention. 
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• Some activities similar to those espoused by the Diplomas Now model were 
taking place at a majority of the study schools prior to the evaluation, but 
some essential components of the Diplomas Now model were unlikely to be 
in place at the study schools. 

• No statistically significant differences were found between DN and non-DN 
schools in any of the survey measures of prestudy programs, further suggest- 
ing that the two groups were similar prior to the start of the evaluation. 


Recruitment of School Districts and Schools 

The Diplomas Now model is designed to strengthen urban middle and high schools that 
struggle with large populations of students at risk of dropping out. The recruitment of school 
districts and schools for the Investing in Innovation (i3) Diplomas Now validation grant reflects 
this design. Diplomas Now focused recruitment efforts on many of the largest school districts in 
the country. Diplomas Now staff members and district administrators typically targeted schools 
within these districts with high numbers of high-needs students, as measured by academic 
achievement. School districts approached about participation in the evaluation and the oppor- 
tunity to have the Diplomas Now model implemented in a subset of their schools were often 
places where two of the Diplomas Now partner organizations, Communities In Schools and 
City Year, already had local presences and existing relationships with district administrators. At 
least one of those organizations had a presence in every district approached about participation. 
The presence of one or more of the Diplomas Now partners meant that the recruitment effort 
could build on existing relationships with district administrators and that the implementation of 
the Diplomas Now model would not be delayed while partner organizations built up their local 
infrastructure. 

The start-up of Diplomas Now’s implementation was staggered over two years, and 
school districts and schools were recruited and began participating in the evaluation in two 
waves. The first wave of schools entered the study at the start of the 2011-2012 school year. 
Twenty-two schools — 9 high schools and 13 middle schools — were recruited in this wave. 
As schools from a particular district were recruited and agreed to participate in the study, they 
were assigned to a lottery with one or more schools from the same district, including the same 
grade levels of students (middle or high schools), and with similar state test scores and 
preprogram demographic characteristics (percentage of minority students, special education 
students, students for whom English was a second language, and students of low socioeco- 
nomic status). Via these lotteries, schools were randomly assigned to participate in the 
Diplomas Now model (DN schools) or to continue with “business as usual” (non-DN 
schools). The business-as-usual condition at the non-DN schools was either the continuation 
of existing school programs or implementation of another reform strategy of the district’s or 
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school’s choosing. 1 The non-DN schools serve as a point of comparison, representing how 
schools assigned to the Diplomas Now group (and their students) would have performed had 
they not had the opportunity to implement the intervention. Nine separate lotteries across 
six school districts were held during the first wave, resulting in 12 DN schools and 10 non- 
DN schools. 

The second wave of schools began participation in the evaluation at the start of the 
2012-2013 school year. Forty more schools were recruited for this wave, 20 middle schools and 
20 high schools. These schools were randomly assigned through 13 separate lotteries across 
nine districts resulting in 20 DN schools and 20 non-DN schools. 

The end result is a sample of 62 total schools from 1 1 participating school districts 
spread across the country, with 32 DN schools and 30 non-DN schools to be included in 
analyses answering the main research questions exploring attendance, behavior, and course 
performance for sixth- and ninth-grade students across middle and high schools combined. In 
some cases, middle school and high school findings will be explored separately; for example, 
the impact of Diplomas Now on ninth-grade completion will only be examined at the high 
schools. There are 33 middle schools and 29 high schools in the study. 


Challenges to Recruitment 

Although site recruitment resulted in a large sample of schools, there were two conditions that 
affected the recruitment process. First, the recruitment of schools and districts for the i3 Diplo- 
mas Now validation grant was affected by the grant timeline and by the need to collect a large 
enough sample for the required evaluation. Since the grant has a strict five-year time limit, 
recruitment needed to move at a particularly fast pace. The grant was awarded in the fall of 
2010 and a first wave of schools needed to be recruited and begin implementation activities 
before the 2011-2012 school year. At the same time, a large number of schools needed to be 
recruited to meet the requirements of the grant and the needs of the study. 2 

Given the tight grant timeline, the Diplomas Now partners restricted themselves to 
school districts where Communities In Schools and City Year already had a local presence. 
Focusing on these districts meant that the recruitment effort could build on existing rela- 
tionships with district administrators, and that implementation would be better positioned 
for a fast start. The local presence of these two Diplomas Now partners meant that it would be 


The reform strategies and programs taken up by non-DN schools will be described in future reports. 

2 To meet the needs of the experiment, almost double the number of schools need to participate in the study 
as are planned to implement the program. This allows for random assignment of similar schools into DN and 
non-DN groups. 
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easier to identify and recruit staff members to work in schools assigned to implement the 
Diplomas Now model, and that infrastructure would already be in place for those Communities 
In Schools and City Year staff members. However, this expectation did mean that some districts 
interested in bringing Diplomas Now to their schools were not pursued by Diplomas Now as 
participants in the evaluation, which contributed both to bringing in fewer schools in the first 
wave (only 22 of the 62 schools were in the first wave of schools to join the study) and extend- 
ing the recruitment timeline into the summer months, restricting the time for planning and staff 
training at some schools. Despite this limitation, it may have been wise to focus on districts with 
Communities In Schools and City Year presences. In the two districts participating in the 
evaluation where one of those two partners did not already have a presence, start-up proved to 
be more difficult. 

Furthermore, the need to recruit so many schools in such a short time made it harder to 
gain the school-level commitment needed for smooth implementation at the start of the school 
year. Particularly in recruiting the first wave of schools, Diplomas Now partners did not have 
enough time to communicate with and attract the support of school leaders. In most cases 
Diplomas Now staff members had to reach out to leaders at schools to begin getting them 
invested in the model after district administrators had already committed their schools to 
participating in the evaluation. During the second wave of recruitment, the Diplomas Now 
partners had more opportunities to gain school-level investment before schools committed to 
participate in the project. Nonetheless, even in the second wave it was not always possible to 
comiect with school leaders early enough. Thus, the need to recruit a large number of schools 
within the first two years of the i3 grant created challenges in securing school-level investment. 

Second, the complexities of getting the necessary funding lined up to implement the 
Diplomas Now model sometimes limited the total number of schools in a district that could 
participate, or extended the time it took to complete recruitment in a district. This challenge was 
probably exacerbated by the cuts to state education funding experienced by most school districts 
during this time period due to the residual effects of the recession. 3 On average, the start-up year 
costs to implement Diplomas Now totaled about $491,000. Typically, Talent Development 
Secondary’s costs were $255,000, which covered curricular materials, the school transformation 
facilitator placed at the school, and technical assistance from mathematics and English/language 
arts instructional facilitators and from a school and student support services facilitator (who 
worked across all implementing schools in a district). City Year’s costs were about $129,000 for 
the City Year team of near peers and an on-site program manager. Communities In Schools’ 
costs averaged $107,000 for the Communities In Schools site coordinator, service provision, 
and infrastructure support. School-level instructional coaches were often an in-kind contribution 


3 Leachman and Mai (2013). 
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from the school districts. In the few cases where they were not, those costs were included in 
Talent Development Secondary’s costs (and are reflected in the Talent Development Secondary 
average cost figure above). 

Diplomas Now implementation costs were covered in almost all sites by a combination 
of Diplomas Now i3 grant money, other funds from the Diplomas Now partners, and district 
funds. On average, i3 funding covered about $314,000 of the first-year costs of an implement- 
ing school. District contributions, in a few cases bolstered by federal School Improvement 
Grant (SIG) awards, averaged $141, 000. 4 Other, privately raised funds accounted for about 
$36,000 per school. 


Description of Participating School Districts 

Recruitment for the Diplomas Now i3 evaluation targeted large and midsize urban school 
districts with high concentrations of high-needs students. Out of the 1 1 school districts partici- 
pating in the evaluation, 5 are among the top 20 largest school districts in the country according 
to the National Center for Education Statistics’ review of data from the 2007-2008 school year, 
and all but one are in the top 100 largest school districts. 5 Table 2.1 displays the characteristics 
of the 1 1 school districts participating in the Diplomas Now i3 evaluation during 2010-2011, the 
school year prior to the start of the evaluation. As noted, the school districts participating in the 
evaluation are large, with an average of 426 schools and approximately 262,000 students per 
district. In comparison, the 100 largest school districts have an average of 169 schools and 
approximately 112,000 students. 6 The school districts participating in the evaluation also have 
large populations of minority students, with approximately 83 percent black or Hispanic 


4 The initial plan for covering the costs of implementing the Diplomas Now model counted on the use of 
federal School Improvement Grant (SIG) funds for school turnaround or transformation at most school sites. 
As recruitment proceeded, it became clear that SIG funding was not likely to complete the funding picture in 
many districts. Although during recruitment the Diplomas Now partners mentioned to school districts that the 
federal i3 and SIG programs were complementary and potentially aligned, federal and state governmental 
agencies did not often communicate this alignment. (SIG funds were disbursed from the federal government to 
state governments. State governments were responsible for the distribution of those funds to school districts.) 
For example, i3 validation grantees like the Diplomas Now partnership had already been vetted through a 
competitive process as having evidence-based programs; thus public agencies could have promoted schools 
intending to implement those programs as a potentially wise investment of SIG funds. In addition, the 
Diplomas Now staff offered to help complete SIG applications on behalf of DN schools, but this offer was not 
pursued by some districts, and in others the timing of SIG funding applications and awards did not align with 
Diplomas Now implementation timelines. 

5 Plotts and Sable (2010). 

6 Plotts and Sable (2010). 


21 



Diplomas Now 
Table 2.1 

Characteristics of Diplomas Now Study School Districts (2010-2011) 


Characteristic 

Average 

Number of schools 

426 

Number of students 

262,426 

Race/ethnicity of students (%) 
Black 

46.0 

Hispanic 

36.6 

Asian 

4.6 

White 

11.3 

Other 

1.3 

Number of students per teacher 

15.8 

English language learner students (%) 

11.3 

Special education students (%) 

14.9 

Sample size 

11 


SOURCE: U.S. Department of Education, National Center for Educational Sciences, Common 
Core of Data, "Local Education Agency Universe Survey," 2010-201 1. 

students (46 percent black and 37 percent Hispanic). This is approximately 20 percentage points 
higher than the average across the 100 largest school districts. 7 The participating school districts 
also have a higher percentage of special-education students than the 100 largest school dis- 
tricts. 8 The districts participating in the evaluation represent many regions across the country 
including the Northeast, Southeast, Midwest, South, and West. 


Description of Participating Schools 

Diplomas Now targets middle and high schools believed to be most able to benefit from the 
Diplomas Now model, typically schools serving large populations of high-needs students. 
These schools are generally high-poverty, urban schools with large populations of minority 
students. This is reflected in Table 2.2, which compares schools participating in the evaluation 
with other schools in the participating school districts, and with the national average. In 


7 On average, 63 percent of students are black or Hispanic (26 percent black and 37 percent Hispanic) 
across the 100 largest school districts. Plotts and Sable (2010). 

8 Eleven percent of students at the 100 largest school districts had special needs. Plotts and Sable (2010). 
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Diplomas Now 
Table 2.2 


Characteristics of Diplomas Now Study Schools, Other Schools in 
Study Districts, and Average Schools in the United States (2010-2011) 


Characteristic 

Study 

Schools 

Other Schools in 
Study Districts 3 

Average U.S. 
Schools’ 5 

Panel A: middle schools 




Eligible for Title I (%) 

100.0 

92.9 

72.3 ** 

Students eligible for free/reduced-price lunch (%) 

83.4 

79.1 

50.2 ** 

Race/ethnicity of students (%) 




Black 

62.4 

46.1 * 

16.4 ** 

Elispanic 

31.3 

39.6 

29 2 ** 

Asian 

2.0 

3.9 

3.9 

White 

3.6 

9.2 * 

56.7 ** 

Other 

0.7 

1.3 

3.8 * 

Gender of students (%) 




Male 

52.1 

50.9 

51.2 

Average number of students 

588.6 

419.1 * 

518.6 

Average number of students in grade 6 L 

186.6 

135.4 

171.0 

School setting d (%) 




City 

87.9 

88.6 

24.7 ** 

Suburb 

12.1 

10.5 

29.0 * 

Town 

0.0 

0.1 

14.8 * 

Rural area 

0.0 

0.8 

31.4 ** 

Average number of full-time teachers 

42.5 

38.0 

38.4 

Sample size 

33 

898 

16,555 


(continued) 


comparison with the national average, the schools participating in the evaluation serve more 
low-income students and minority students, and are more likely to be located in urban areas. All 
of the schools participating in the evaluation are eligible for Title I, compared with 54 percent of 
high schools and 72 percent of middle schools across the country. 9 On average, 73 percent of 


9 Title I is a program created by the United States Department of Education to distribute funding to schools 
and school districts with high percentages of low-income families. To be eligible for school-wide Title I, at 
least 40 percent of a school’s students must be from low-income families. 
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Table 2.2 (continued) 


Characteristic 

Study 

Schools 

Other Schools in 
Study Districts 3 

Average U.S. 
Schools’ 5 

Panel B: high schools 

Eligible for Title I (%) 

100.0 

93.1 

54.1 ** 

Students eligible for free/reduced-price lunch (%) 

73.2 

72.0 

41.0 ** 

Race/ethnicity of students (%) 

Black 

58.1 

38.3 ** 

16.3 ** 

Hispanic 

32.7 

46.5 * 

16.2 ** 

Asian 

4.7 

4.7 

4.2 

White 

4.0 

8.9 * 

60.3 ** 

Other 

0.6 

1.6 

3.0 * 

Gender of students (%) 

Male 

52.3 

49.6 

50.9 * 

Average number of students 

1,438.8 

1,291.3 

1,204.5 

Average number of students in grade 9 

445.3 

379.3 

330.2 ** 

School setting d (%) 

City 

82.8 

85.4 

22.3 ** 

Suburb 

17.2 

13.2 

30.9 

Town 

0.0 

0.0 

17.4 * 

Rural area 

0.0 

1.4 

29.4 ** 

Average number of full-time teachers 

78.9 

68.8 

70.7 

Average promoting power 3 

56.0 

65.8 * 

80.9 ** 

Sample size 

29 

349 

9,236 


(continued) 


students in study high schools and 83 percent of students in study middle schools are eligible for 
free or reduced-price lunches, compared with national averages of 41 percent and 50 percent, 
respectively. Ninety-one percent of students in participating high schools and 94 percent of 
students in participating middle schools are black or Hispanic, while nationally only 33 percent 
of high school students and 36 percent of middle school students are black or Hispanic. Over 
three-quarters of study schools are identified as being located in urban areas, while nationally 
only about a quarter of schools are located in urban areas. The high schools participating in the 
study also have lower promoting power than the national average, suggesting that they are less 
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Table 2.2 (continued) 


SOURCES: U.S. Department of Education, National Center for Educational Sciences, Common Core of 
Data, "Public Elementary/Secondary School Universe Survey Data," 2010-2011 and 2007-2008. 

NOTES: Four study schools, 22 schools in the "other schools in study districts" sample, and 2,798 
schools in the "average U.S. schools" sample are missing data for "average number of students in 
grade 6" because they did not include sixth grades during the baseline year. Two middle schools and 
four high schools in the study schools sample are missing data on "average number of full-time 
teachers." One hundred and nineteen middle schools and 93 high schools are missing these data in the 
"other schools in study districts" sample, and 1,697 middle schools and 828 high schools are missing 
these data in the "average U.S. schools" sample. Nine middle schools and 13 high schools in the "other 
schools in study districts" sample are also missing data for "students eligible for free/reduced-price 
lunch," and 36 middle schools and 26 high schools in the "average U.S. schools" sample are missing 
data for this variable. 

A two-tailed t-test was applied to differences between the program group and the comparison 
groups. Statistical significance levels are indicated as follows: ** = 1 percent; * = 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a "Other schools in study districts" include all other nonstudy schools in the same school districts as 
the Diplomas Now study schools that met the "average U.S. schools" criteria below. 

b "Average U.S. schools" include non-Diplomas Now study middle schools that during 2010-201 1 had 
more than 25 total seventh-grade students and 25 eighth-grade students and non-Diplomas Now study 
high schools that had more than 100 total ninth-grade students, and during 2007-2008 had at least 100 
students in the ninth grade. "Average U.S. schools" are also defined as "regular" schools by the 
Common Core of Data, and are located within the 50 U.S. states and the District of Columbia. 

c Four schools in the "study schools" sample did not have sixth grades during the 2010-2011 school 
year and are excluded from this analysis. Schools with fewer than 25 sixth-grade students were also 
excluded from the other two samples. 

d A "city" is defined as a territory inside an urbanized area and inside a principal city with a 
population greater than 100,000. A "suburb" is defined as a territory outside of a principal city and 
inside an urbanized area with a population of less than 250,000. 

e "Promoting power" is calculated as the ratio of twelfth-grade students in 2010-201 1 to ninth-grade 
students in 2007-2008. 


likely to move students from ninth through twelfth grade on time. 10 The average promoting 
power for study high schools shows that 56 percent of ninth-grade students reach twelfth grade 
in three years, while the national average is over 80 percent. 

As discussed above, the school districts participating in the study are generally large ur- 
ban districts with high percentages of minority students. Schools participating in the evaluation 
are representative of the schools in these districts, with similarly high rates of eligibility for Title 
I and free or reduced-price lunches, similarly high rates of minority students, and slightly higher 


10 “Promoting power” is calculated as the ratio of twelfth-grade students in 2010-2011 to ninth-grade stu- 
dents in 2007-2008. 
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percentages of black students. Since the Diplomas Now program sought to recruit schools 
serving students most at risk of dropping out, it is not surprising that high schools participating 
in the evaluation have a lower promoting power (56 percent) than the average across all high 
schools in participating districts (66 percent). 

Table 2.3 compares DN and non-DN high schools and middle schools prior to program 
implementation (during the 2010-201 1 school year) in a variety of dimensions, and shows that 
DN and non-DN schools are similar for most of them. An overall test of the differences between 
the DN and non-DN schools for all the baseline characteristics that appear in the table confirms 
that the groups are not systematically different from each other, as would be expected following 
random assignment. These baseline findings indicate that the non-DN group provides a con- 
vincing representation of what would have happened in the DN schools had they not imple- 
mented the intervention. 


Programs at All Schools Prior to Diplomas Now Implementation 

The Diplomas Now model brings together an intensive set of instructional, organizational, 
data-related, student-related, and professional-development components and is implemented 
at a school in a systematic format with a team of auxiliary staff members and volunteers. 
These model components and additional staff members enable the implementation of whole- 
school reforms, targeted interventions for struggling students, and intense interventions for 
the neediest students. To illuminate the structures, curricula, programming, services, and 
climate in the study schools prior to the implementation of Diplomas Now, and to determine 
whether any features similar to those of the Diplomas Now program existed in the study 
schools prior to the evaluation, teachers and administrators (principals and assistant princi- 
pals) were surveyed early in the first year of the evaluation and asked about their experiences 
at the study schools during the prior school year. 1 1 Eighty-two percent of administrators and 
78 percent of teachers participated in the survey. 12 Even given these high response rates in 
general, due to missing data 1 6 schools are not included in the analyses of the administrator 
surveys and 6 schools are not included in the analyses of the teacher surveys. Most schools 


1 1 The timing of school recruitment, which happened in the spring and summer prior to implementation, 
made it impossible to survey teachers before implementation began. Surveys were administered in fall 2011 for 
first wave of schools and fall 2012 for the second wave of schools. Most of the survey items asked respondents 
about their experiences during the prior school year. Only teachers and administrators who were present at the 
study school during the previous school year were asked to answer these items. 

12 See Appendix B for detailed information on survey response rates. 
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Diplomas Now 
Table 2.3 


Characteristics of Study Schools, Overall and by School Level 



DN 

Non-DN 

Estimated 


Characteristic 

Schools 

Schools 

Difference 

P- Value 

Panel A: all schools 





Students eligible for free/reduced-price lunch (%) 

80.6 

79.5 

1.1 

0.620 

Race/ethnicity of students' 1 (%) 





Black 

58.7 

64.4 

-5.7 

0.202 

Hispanic 

34.4 

25.9 

8.5 * 

0.018 

Asian 

3.1 

4.4 

-1.3 

0.395 

White 

3.2 

4.6 

-1.4 

0.207 

Other 

0.6 

0.7 

-0.1 

0.814 

Gender of students (%) 





Male 

51.9 

52.2 

-0.3 

0.904 

Teacher experience (%) 





0-4 years teaching 

25.6 

19.2 

6.4 ** 

0.008 

5 or more years teaching 

74.4 

80.8 

-6.4 ** 

0.008 

Teacher education (highest degree) (%) 





Bachelor's 

34.8 

34.3 

0.5 

0.834 

Master's or higher 

65.2 

65.7 

-0.5 

0.834 

Teacher certification' 1 (%) 





Advanced certificate or National Board Certification 

16.6 

15.2 

1.3 

0.587 

Professional, regular, or standard state certificate 

75.8 

79.2 

-3.4 

0.157 

Temporary, probationary, or other certificate 

7.7 

5.6 

2.1 * 

0.049 

Sample size 

32 

30 



Panel B: middle schools 





Average school enrollment 





Grade 6 

182.3 

212.3 

-30.1 

0.227 

Grade 7 

189.1 

248.5 

-59.5 

0.087 

Grade 8 

188.2 

251.6 

-63.4 

0.093 

Students proficient on state test (%) 





Grade 6 Math 

42.5 

41.5 

1.0 

0.800 

Grade 6 English/language arts 

37.2 

38.2 

-1.0 

0.790 

Grade 7 Math 

44.3 

43.3 

1.0 

0.758 

Grade 7 English/language arts 

43.4 

41.3 

2.0 

0.628 

Grade 8 Math 

45.4 

42.8 

2.6 

0.477 

Grade 8 English/language arts 

48.1 

43.7 

4.4 

0.251 

Sample size 

17 

16 




(continued) 
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Table 2.3 (continued) 


Characteristic 

DN 

Schools 

Non-DN 

Schools 

Estimated 

Difference 

P- Value 

Panel C: high schools 

Average school enrollment 

Grade 9 

453.9 

457.0 

-3.1 

0.941 

Grade 10 

390.0 

408.9 

-18.9 

0.645 

Grade 11 

322.5 

315.6 

6.9 

0.849 

Grade 12 

312.1 

301.6 

10.6 

0.781 

Students proficient on state test b (%) 

English/language arts 

43.1 

50.3 

-7.2 * 

0.047 

Math 

40.4 

45.1 

-4.7 

0.204 

Average promoting power c (%) 

57.6 

52.6 

5.0 

0.349 

Sample size 

15 

14 




SOURCES: U.S. Department of Education, National Center for Educational Sciences, Common Core 
of Data, "Public Elementary/Secondary School Universe Survey Data," 2010-2011, state 
achievement test data, 2010-2011. Baseline surveys of administrators (principals and assistant 
principals) and teachers administered during the school year of 2011-2012 (for wave 1 schools) and 
2012-2013 (for wave 2 schools). 

NOTES: For teacher experience, teacher education, and teacher certification, data for one DN and 
two non-DN schools are missing. Sixth-grade average school enrollment data and sixth-grade math 
and English/language arts state test proficiency data are missing for two DN and two non-DN 
schools that did not include sixth grades during the baseline year. 

Difference estimates are regression-adjusted, controlling for the blocking of random assignment. 

A two-tailed t-test is used for all statistical tests presented in this table. Statistical significance levels 
are indicated as follows: ** = 1 percent; * = 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a A chi-squared test was used to determine whether there is a difference between DN and non-DN 
schools for the combined race/ethnicity categories and teacher certification categories. The p-value 
for the combined race/ethnicity categories was 0.124 and the p-value for the teacher certification 
categories was 0.037. 

b High school state achievement proficiency information is taken from the statewide math and 
English/language arts tests given in the highest grade. 

c "Promoting power" is calculated as the ratio of twelfth-grade students in 2010-2011 to ninth- 
grade students in 2007-2008. 

were removed from the analyses because none of the administrators or teachers who participat- 
ed in the survey were present at the school the year before. 13 


l 3 The 16 schools dropped from the administrator survey analyses include: 1) 8 schools where none of the 
administrators who responded to the survey were present during the prior year, 2) 5 schools that were in a 
random assignment block where all of the DN schools or all of the non-DN schools were dropped due to 

(continued) 
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Tables 2.4 through 2.7 compare the percentages of administrators and teachers in the 
DN and non-DN schools who responded affirmatively about their schools having specific 
structural, curricular, or cultural features similar to those that make up the Diplomas Now 
model. Each table is aligned with one of the Four Pillars of the Diplomas Now model: Teacher 
Teams and Small Learning Communities (Table 2.4), Curriculum and Instruction with Profes- 
sional Development (Table 2.5), Tiered Student Supports (Table 2.6), and a Can-Do Culture 
and Climate (Table 2.7). 

These tables tell two stories. First, they compare DN and non-DN schools prior to Di- 
plomas Now implementation to assess whether there were any major differences between them 
in certain important domains of the Diplomas Now program. Across administrator and teacher 
responses, there are no statistically significant differences between the DN and non-DN schools, 
suggesting that the two sets of schools were similar at the start of the evaluation in their adop- 
tion of structural and programmatic features like those of the Diplomas Now model. 

Second, they also reveal whether, in general, features similar to those of the Diplomas 
Now program existed at the study schools prior to Diplomas Now implementation. Overall, 
these tables suggest that some structures, practices, and processes similar to those in the Diplo- 
mas Now model did take place in a majority of study schools prior to the evaluation. That is, the 
schools in the study were not “blank slates,” but were in many cases attempting some curricular, 
structural, or programmatic refonns similar to those associated with the Diplomas Now model. 
For example, a majority of study school administrators reported that services for students were 
offered prior to program implementation, including individual and group counseling, enrich- 
ment and academic services provided during an extended school day, and credit-recovery or 
remedial courses for struggling high school students. Still, since most of the survey items only 
ask generally about school structures and components and do not detail the frequency or 
duration of activities, it is hard to judge the scale and quality at which these components were 
being implemented, or whether they were implemented in a fomiat similar to that of the 
Diplomas Now model. It is also not clear from these findings whether schools were implement- 
ing these program components as part of a coordinated and systematic school-wide reform like 
the Diplomas Now model. 


missing data (since there was no comparison, the entire random assignment block was dropped); and 3) 3 
schools where data were lost after collection. The 6 schools dropped from the teacher survey analyses include: 
1)2 schools where none of the teachers who responded to the survey were present during the prior year, 2) 1 
school that was in a random assignment block where all of the DN schools or all of the non-DN schools were 
dropped due to missing data, and 3) 3 schools where data were lost after collection. 
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Diplomas Now 
Table 2.4 


Pillar I: Teacher Teams and Small Learning Communities, 
Baseline Administrator and Teacher Survey Responses 


Survey Item 

DN Non-DN Estimated 

Schools (%) Schools (%) Difference 

P- Value 

Panel A: administrators 





My school was subdivided into distinct 

organizational units (for example, schools-within-a-school, 

thematic academies or small learning communities). 3 

56.7 

65.4 

-8.7 

0.507 

The majority of courses were extended periods 
(70 or more minutes). 

61.3 

68.1 

-6.8 

0.478 

Classes were organized in a 4x4 +1 block 
schedule (4 extended learning periods and 
1 enrichment elective), (high school only)' 0 

40.2 

31.9 

8.3 

0.586 

Sample size 

24 

22 



Panel B: teachers 





Most core classes taught were extended periods. 

52.4 

61.0 

-8.6 

0.299 

Core teachers reported collaborating with an 
interdisciplinary team of other teachers who shared 
the same group of students weekly. 

32.7 

38.8 

-6.1 

0.230 

Math and English/language arts teachers reported 
participating in a professional learning community with 
teachers from the same subject areas weekly. 

34.8 

43.5 

-8.6 

0.119 

Sample size 

30 

26 




SOURCES: Baseline surveys of administrators (principals and assistant principals) and teachers 
administered during the school years of 20 11 -20 12 (for wave 1 schools) and 2012-2013 (for wave 2 
schools). Due to missing data, 6 schools have been dropped from the analysis of teacher survey data and 
1 6 schools have been dropped from the analysis of administrator survey data. 

NOTES: The values in the column labeled "DN schools" are the averages across schools of the percentage 
of administrators or teachers at each DN school who answered each item in the affirmative. The values in 
the column labeled "non-DN schools" are the regression-adjusted averages. The estimated differences are 
regression-adjusted using ordinary least squares, controlling for blocking of random assignment by 
school. 

"Core" academic areas are identified as math, English/language arts, sciences, and social studies. 

A two-tailed t-test was applied to differences between the program group and the comparison group. 
Statistical significance levels are indicated as follows: ** = 1 percent; * = 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a One additional school is missing data for this item. 

b There are 12 DN high schools and 1 1 non-DN high schools. 
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Diplomas Now 
Table 2.5 


Pillar II: Curriculum and Instruction with Professional Development, 
Baseline Administrator and Teacher Survey Responses 


Survey Item 

DN 

Schools (%) 

Non-DN Estimated 
Schools (%) Difference 

P- Value 

Panel A: administrators 





Students participated in college preparation curricula 
in core academic areas, (high school only ) a 

94.2 

98.8 

-4.6 

0.206 

Struggling students were offered opportunities 
for course recovery or were provided with 
remedial classes to enable them to catch up 
to their peers, (high school only) “ 

96.5 

98.1 

-1.6 

0.592 

Principals had the opportunity to participate in a 
principal/leader support network. 

84.4 

92.1 

-7.7 

0.205 

Sample size 

24 

22 



Panel B: teachers 





Math and English/language arts teachers reported 
receiving weekly instructional coaching from a 
school teacher (a peer coach, mentor, or math or 
English/language arts facilitator). 

43.1 

51.2 

-8.1 

0.061 

Math and English/language arts teachers reported 
that students in their classes were engaged in 
different learning activities at the same time at least 
weekly. 

65.8 

60.7 

5.0 

0.174 

Math and English/language arts teachers reported that 
their lessons included applications to real-life issues 
at least weekly. 

81.5 

79.6 

1.9 

0.546 

Sample size 

30 

26 




(continued) 


Some important components of the Diplomas Now model were less likely than others 
to be reported in a majority of schools prior to program implementation. For example, fewer 
than half of the teachers reported participating in interdisciplinary teams and professional 
learning communities, and few teachers reported having extensive volunteer assistance in their 
classrooms like that offered by the City Year corps members in the Diplomas Now model. 

On average, a majority of the administrators surveyed indicated that activities, struc- 
tures, and practices similar to those in the Diplomas Now model were going on at their schools 
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Table 2.5 (continued) 


SOURCES: Baseline surveys of administrators (principals and assistant principals) and teachers 
administered during the school years of 201 1-2012 (for wave 1 schools) and 2012-2013 (for wave 2 
schools). Due to missing data, 6 schools have been dropped from the analysis of teacher survey data 
and 1 6 schools have been dropped from the analysis of administrator survey data. 

NOTES: The values in the column labeled "DN schools" are the school-level averages of the 
percentage of administrators or teachers at each DN school who answered each item in the affirmative. 
The values in the column labeled "non-DN schools" are the regression-adjusted averages. The 
estimated differences are regression-adjusted using ordinary least squares, controlling for blocking of 
random assignment by school. 

"Core" academic areas are identified as math. English/language arts, sciences, and social studies. 

A two-tailed t-test was applied to differences between the program group and the comparison 
group. Statistical significance levels are indicated as follows: ** = 1 percent; * = 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
a There are 12 DN high schools and 1 1 non-DN high schools. 


prior to Diplomas Now implementation. 14 This is most noticeable in Pillar II (Table 2.5), where 
almost all administrators at a school, on average, indicated that all students at their high schools 
participated in college prep curricula in core content areas, and that struggling students were 
offered opportunities for course recovery or were provided remedial classes to enable them to 
catch up to their peers. Although it is not possible to gauge the quality of the preevaluation 
curricula at these schools, it is important to note that most schools were already making efforts 
to focus on a college prep curriculum and offer remedial course work to students falling behind. 

It is notable, however, that in Table 2.4, fewer than half of the administrators at DN 
high schools and non-DN high schools suggested that their schools had classes organized in a 
4x4 +1 block schedule that allowed for four extended learning periods for core subjects and one 
enrichment elective course. The Diplomas Now team views this organizational structure as 
essential to successful curricular delivery, particularly for remedial course work in the ninth 
grade. This suggests that although study high schools may already have had some remedial 
course offerings for their students, they were probably organized differently from those in 
Diplomas Now schools, and they may not have offered the same kind of accelerated learning as 
the double-dose offerings of math and English/language arts provided to Diplomas Now 
students testing more than one year behind grade level in these subjects. 


14 To capture better how administrators responded across the entire survey, the research team created a 
composite measure that includes all 11 administrator measures in Tables 2.4 through 2.7 where both middle 
and high school administrators participated. On average across all DN and all non-DN schools, a majority of 
administrators at each school responded to 69 percent of the 1 1 items affirmatively. 
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Diplomas Now 
Table 2.6 


Pillar III: Tiered Student Supports, 

Baseline Administrator and Teacher Survey Responses 


Survey Item 

DN Non-DN Estimated 

Schools (%) Schools (%) Difference 

P- Value 

Panel A: administrators 





Individual, group, and family counseling services 
were all available at the school. 

60.1 

67.1 

-7.0 

0.566 

Students were provided with clothing, school 
supplies, food, and health and wellness activities 
as needed. 

62.9 

79.6 

-16.7 

0.102 

Students participated in academic and enrichment 
programs through an extended school day. 

72.6 

68.7 

3.9 

0.727 

A school-wide behavior management system existed 
at the school. 3 

35.9 

37.7 

-1.8 

0.871 

Sample size 

24 

22 



Panel B: teachers 





Interdisciplinary teams reviewed data on individual 
students' academic progress, course performance, 
attendance, and behavior to identify students' needs 
and determine appropriate responses. 

49.9 

52.5 

-2.5 

0.573 

Math and English/language arts teachers reported 
volunteers worked with students in at least some of 
their classes on a daily basis. 

15.5 

10.2 

5.3 

0.226 

Core teachers reported arranging for intensive 
support and interventions for students by making 
appropriate referrals as needed. 

80.7 

76.6 

4.1 

0.164 

Sample size 

30 

26 




(continued) 


As shown in Table 2.6, most school administrators did report that prior to the evalua- 
tion, schools were offering students counseling and other health and wellness services and some 
academic and enrichment programming before or after normal school hours. It is worth noting 
that Communities In Schools was being fully implemented in at least five study schools (two 
program and three control schools) during the year prior to Diplomas Now implementation. The 
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Table 2.6 (continued) 


SOURCES: Baseline surveys of administrators (principals and assistant principals) and teachers 
administered during the school years of 2011-2012 (for wave 1 schools) and 2012-2013 (for wave 2 
schools). Due to missing data, 6 schools have been dropped from the analysis of teacher survey data 
and 16 schools have been dropped from the analysis of administrator survey data. 

NOTES: The values in the column labeled "DN schools" are the school-level averages of the 
percentage of administrators or teachers at each DN school who answered each item in the affirmative. 
The values in the column labeled "non-DN schools" are the regression-adjusted averages. The 
estimated differences are regression-adjusted using ordinary least squares, controlling for blocking of 
random assignment by school. 

"Core" academic areas are identified as math. English/language arts, sciences, and social studies. 

A two-tailed t-test was applied to differences between the program group and the comparison 
group. Statistical significance levels are indicated as follows: ** = 1 percent; * = 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a This measure is the average of the following items: teachers consistently enforced rules for student 
behavior in their classrooms; teachers helped maintain discipline in the entire school, not just their 
own classrooms (for example, by helping students transition between classes); the school had an 
effective system for providing positive reinforcement to students who met behavior expectations; the 
school had an effective system for responding to problem behaviors; students received instruction and 
guidance on school discipline policies and procedures; educators conducted functional assessments 
and implemented individual behavior plans; and students with frequently disruptive behavior received 
daily check-ins with adults to monitor their progress in meeting behavior goals. 


organization also had relationships with at least six other schools (three program and three 
control schools) where program components were being provided to subsets of students. 15 

Just over a third of administrators at DN and non-DN schools reported that their schools 
had school-wide behavior management systems with components similar to those the Diplomas 
Now model promotes. Managing student behavior and offering assistance to students with 
social (as well as academic) needs is of particular importance to the Diplomas Now model. 

Finally, although administrators reported that their schools might include some compo- 
nents similar to those in Diplomas Now, as shown in Table 2.7 fewer than half of them at DN 
and non-DN schools reported any kind of externally developed reform model being used prior 
to Diplomas Now implementation. Still, most administrators in DN and non-DN schools 
indicated that they had staff members designated to coordinate school reform efforts, intensive 
individual interventions, and community resources. 


The study team was unable to confirm the baseline presence or absence of Communities In Schools in 
12 of the 62 study sample schools. 
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Diplomas Now 
Table 2.7 


Pillar IV: Can-Do Culture and Climate, 
Baseline Administrator and Teacher Survey Responses 


Survey Item 

DN Non-DN Estimated 

Schools (%) Schools (%) Difference 

P- Value 

Panel A: administrators 





My school used an externally developed school 
reform model. 

49.7 

46.1 

3.6 

0.739 

A designated adult at the school was in charge of 
coordinating school reform efforts. 

73.0 

63.4 

9.6 

0.385 

A designated adult at the school was in charge of 
coordinating intensive interventions and 
community resources. 2 

69.3 

75.8 

-6.5 

0.481 

Parents/guardians and community members were 
involved in school activities, helped make school 
decisions, and supported school reform efforts. 

46.7 

60.1 

-13.4 

0.200 

Sample size 

24 

22 



Panel B: teachers 





Teachers were involved in decisions about school 
policies, supported school reform efforts, and in 
general, had positive morale. 

35.9 

40.4 

-4.5 

0.284 

The environment at the school was conducive to 
teaching and learning. 

53.1 

57.1 

-4.0 

0.418 

Students were respectful to peers and teachers, 
maintained academic integrity, took 
responsibility for their own learning, paid attention, 
and did their schoolwork. b 

25.9 

28.8 

-2.9 

0.363 

Sample size 

30 

26 




(continued) 


In contrast to the administrators surveyed, teachers were generally less likely to indi- 
cate that structures and programs similar to the Diplomas Now model’s components existed at 
their schools prior to Diplomas Now implementation . 16 For example, one important structural 


16 To capture better how teachers responded across the multiple items, the research team created a compo- 
site measure that combines all 12 teacher measures in Tables 2.4 through 2.7. On average across all DN 
schools, a majority of teachers at each school responded affirmatively to 48 percent of the 12 items, and on 

(continued) 
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Table 2.7 (continued) 


SOURCES: Baseline surveys of administrators (principals and assistant principals) and teachers 
administered during the school years of 201 1-2012 (for wave 1 schools) and 2012-2013 (for wave 2 
schools). Due to missing data, 6 schools have been dropped from the analysis of teacher survey data 
and 16 schools have been dropped from the analysis of administrator survey data. 

NOTES: The values in the column labeled "DN schools" are the school-level averages of the 
percentage of administrators or teachers at each DN school who answered each item in the 
affirmative. The values in the column labeled "non-DN schools" are the regression-adjusted averages. 

The estimated differences are regression-adjusted using ordinary least squares, controlling for 
blocking of random assignment by school. 

"Core" academic areas are identified as math. English/language arts, sciences, and social studies. 

A two-tailed t-test was applied to differences between the program group and the comparison 
group. Statistical significance levels are indicated as follows: ** = 1 percent; * = 5 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

a One additional school is missing data for this item. 

b This measure is the average of the following items: students communicated with their peers in a 
respectful manner most of the time; students communicated with me in a respectful manner most of 
the time; students took responsibility for their own learning most of the time; students maintained 
academic integrity on tests or written assignments most of the time; students appeared to take pride in 
their schoolwork most of the time; students put forth effort to understand difficult materials most of 
the time; students appeared to pay attention during my instruction most of the time; students remained 
on task during self-directed activities most of the time; and students completed their assignments most 
of the time. 

component of the Diplomas Now model’s tiered system is for interdisciplinary teams of 
teachers sharing the same group of students to be given consistent and ample time to collabo- 
rate. As shown in Table 2.4, only about a third of teachers reported collaborating at least weekly 
with an interdisciplinary team of teachers who shared the same group of students. Similarly, 
fewer than half the teachers at DN and non-DN schools reported participating at least weekly in 
professional learning communities with teachers who shared their subject areas. For Diplomas 
Now, this is an important structural component of peer coaching and professional development, 
since much of Diplomas Now’s coaching is delivered through these professional learning 
community meetings. As shown in Table 2.5, 43 percent of teachers at DN schools and 51 
percent of teachers at non-DN schools reported receiving weekly instructional coaching prior to 
Diplomas Now. 

While an important feature of the Diplomas Now program is the City Year corps that 
works directly with students inside and outside the classroom, as shown in Table 2.6 few 
teachers reported that volunteers were regularly working in their classrooms prior to Diplomas 
Now. This suggests that the additional classroom support provided by City Year was not a part 

average across all non-DN schools, a majority of teachers at each school responded affirmatively to 5 1 percent 
of the 12 items. The difference between DN and non-DN teachers is not statistically significant. 
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of most study schools prior to the evaluation. In a few cases, City Year was already active in 
schools prior to the full Diplomas Now implementation. City Year was active in 8 of the 62 
study schools (5 program and 3 control schools) during the year before Diplomas Now. 

A majority of teachers did report engaging in some of the practices that Diplomas 
Now promotes for teachers. Teachers at most study schools reported that they were engaging 
students in different learning activities and using lessons with real-life applications on a 
regular basis, as shown in Table 2.5. Teachers also reported that they were arranging for 
intensive support for students by making appropriate referrals (Table 2.6), with 81 percent of 
DN schools’ teachers and 77 percent of non-DN schools’ teachers saying that they did so. 
Still, teachers did not tend to report that their schools had the “can-do culture and climate” 
that Diplomas Now works to promote. Fewer than half of the teachers at DN and non-DN 
schools reported that they were involved in decisions about school policies, supported 
refonns, and felt the morale of the staff was positive. Only a quarter of teachers reported that 
students at their school were respectful to teachers and each other, and took responsibility for 
their learning and schoolwork most of the time. 


Summary 

The Diplomas Now model is designed to meet the needs of urban secondary schools with high 
concentrations of high-needs students, and the schools ultimately recruited for the Diplomas 
Now i3 evaluation fit these characteristics. The DN and non-DN groups of schools are compa- 
rable across a broad range of baseline student characteristics and structural and programmatic 
components. Administrators and teachers reported that prior to the evaluation a majority of 
study schools were making efforts to provide some services similar to those found in the 
Diplomas Now model, suggesting that many study schools were attempting structural, curricu- 
lar, and programmatic reforms prior to Diplomas Now implementation. But it is less clear 
whether those services were of the same scale or quality as those in the Diplomas Now model. 
In some instances, the Diplomas Now teams at DN schools may have been able to capitalize on 
having structural or curricular refomis similar to those of the Diplomas Now model already in 
place. For example, since structural changes such as reorganizing the class and period schedule 
and the length of classes can be particularly difficult for a school to implement quickly, middle 
schools that already had extended periods or high schools that already had 4x4 block scheduling 
may have benefited from the presence of these structures in implementing Diplomas Now. But 
in other instances, having a certain type of reform in place may not have benefited the Diplomas 
Now teams at the DN schools if the reform was not closely aligned with that of the DN model, 
or if it had not been well received or well implemented in past years. Chapter 3 will focus on the 
DN schools during the first year of implementation of the program and discuss how successful 
they were in implementing the features of the Diplomas Now model. 
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Chapter 3 

Fidelity of Implementation of Diplomas Now 


The Diplomas Now model proposes that whole-school organizational and instructional reforms 
and targeted student support create a school climate that is conducive to good teaching and 
learning and that inspires students to attend, behave, and try. This chapter focuses on the 
fidelity of implementation of the Diplomas Now model, drawing on annual data collection 
from the full sample of 32 schools implementing Diplomas Now (DN schools), which consists 
of 17 middle schools and 15 high schools. Fidelity of implementation is defined as the extent 
to which schools implemented the Diplomas Now logic model inputs (see Figure 1.1). The 
research team measured fidelity of implementation in two ways: (1) implementation of all the 
inputs and (2) implementation of the specific aspects of the model deemed most critical by the 
Diplomas Now Implementation Support Team, a team of national-level representatives from 
the three partner organizations. 

This chapter offers several main takeaways and lessons: 

• During the first year of implementation, on average DN schools implemented 
a majority (61 percent) of the program components across the nine program 
inputs. In all, 1 1 1 separate program components were defined and measured 
in collaboration with the Implementation Support Team. Overall, this may 
suggest that implementation of the model had fraction in DN schools by the 
end of the first year. Given the complexity of the model, it is not surprising 
that full implementation did not happen in a single year. 

• However, during the first year of implementation, none of the schools in the 
study were able to adequately implement all of the components that the Im- 
plementation Support Team identified as critical to success. Only half of the 
schools were able to achieve a rating of “moderate” implementation in the 
areas considered most critical by Diplomas Now, while the rest of the 
schools were rated as achieving “low” implementation. None of the schools 
reached the “solid” or “high” implementation rating. 

• DN schools were most successful during the first year in implementing a 
tiered intervention model and adding student support services like tutoring 
and after-school programs into the existing school structure. These are key 
components of Diplomas Now efforts to identify and support students who 
are falling off track. 
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• DN schools were less successful in adopting new curricula and implementing 
peer coaching models. Both of these require the trust and investment of 
school administrators and teachers, and may therefore require additional time 
to implement. 

• At DN schools, the influx of staff members from Talent Development Sec- 
ondary, City Year, and Communities In Schools (that is, the implementation 
of the Integrated On-Site Support input) contributed substantially to the im- 
plementation of other inputs. However, lack of clarity surrounding the roles 
and responsibilities of Diplomas Now staff members may have hindered 
some aspects of implementation during the first year. 

The following sections of this chapter discuss data sources, methods and analyses, and 
the implementation fidelity findings. 


Data Sources 

Primary data for this implementation study came from program staff surveys of all 32 DN 
schools in both wave 1 (2011-2012) and wave 2 (2012-2013). 1 The Diplomas Now program 
staff surveys serve as the basis for determining fidelity and calculating the fidelity metrics. In 
addition, the study team fielded school administrator and teacher surveys in all 32 DN schools 
and visited 7 of them to gather more in-depth, qualitative case-study data through individual 
interviews and focus groups. The school administrator and teacher surveys, and the interview 
and focus-group data, are considered secondary sources that provide additional context to the 
Diplomas Now program staff survey findings. 

The Diplomas Now program staff surveys were developed in collaboration with the 
Implementation Support Team and specifically target the components within each input to 
measure the fidelity of their implementation. The following four surveys were administered to 
program staff members from each of the Diplomas Now partner organizations (see Chapter 1 
for a description of each partner organization) at the culmination of the first implementation 
year: 


1. The Implementation Support Team was asked about each school’s participation in 
the Diplomas Now summer institute, in-service training for program staff members, 
and the kick-off planning sessions and meetings. 


'Diplomas Now staff members collected program documents, databases, and service records to supple- 
ment the program staff surveys. 


40 



2. The school transformation facilitator, who coordinates school reform efforts, an- 
swered questions regarding student learning environments, the school curriculum, 
tiered intervention support, family and community involvement, and the presence 
and activities of curricular support staff members, such as the instructional coaches. 

3. The City Year program manager, who manages the City Year corps at a school, 
was asked about the corps’ staffing and activities, such as tutoring schedules and 
after-school activities. 

4. The Communities In Schools site coordinator, responsible for intensive interven- 
tions, answered questions about the presence, start time, and certification of the site 
coordinator, and about collaboration with Diplomas Now program partners. 

School administrator and teacher sutyeys were also collected during the spring of each 
year. These surveys were administered to principals, assistant principals, and teachers to provide 
supporting data about Diplomas Now implementation from their perspectives. 2 Administrator 
and teacher surveys covered topics related to early program outcomes, such as school climate 
and relationships among various stakeholders, but also covered aspects of school programs 
aligned with the Diplomas Now model, thereby yielding information about Diplomas Now 
implementation and fidelity (see the Data Collection Instrument Supplement for sample school 
administrator and teacher surveys). 3 Administrator surveys were collected from 31 DN schools, 
for a total of 94 respondents. Sixth- and ninth-grade teacher surveys were collected from all 32 
DN schools, for a total of 742 respondents. (See Appendix Table B.l for more details on the 
respondents.) 

Intetyiews and focus groups conducted during qualitative case-study research at seven 
DN schools in four school districts were designed to dig deeper into the processes and interac- 
tions that facilitate or constrain the implementation of the Diplomas Now model. The Imple- 
mentation Support Team helped the research team purposefully select case-study middle 
schools and high schools to represent different geographic regions of the country and diverse 
experiences with Diplomas Now implementation. Interview and focus-group protocols were 
developed and tailored to the role of each respondent group, based on the Diplomas Now logic 


‘Although survey data were collected from all teachers in every school, because Diplomas Now makes a 
priority of services to the sixth and ninth grades, the analyses of staff surveys focused on teachers of those 
grades. Additionally, specific attention was given to sixth- and ninth-grade teachers of core courses (that is. 
English/language arts, math, social studies, and science). 

3 The Data Collection Instrument Supplement to this report, Corrin et al. (2014), is available at 
www.mdrc.org. 
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model (Figure 1.1) and a set of research questions (see the Data Collection Instrument Supple- 
ment for sample protocols): 4 

• How is Diplomas Now implemented across case-study schools? 

• What factors facilitated implementation of the model, and what factors hindered 
implementation? 

• What are the perceived benefits of the Diplomas Now model in case-study schools? 

• What were the drawbacks to implementing the Diplomas Now model? 

• What lessons can be learned from the implementation of Diplomas Now in case- 
study schools? 

Data collected from these seven schools included 49 interviews and 28 focus groups 
with a total of 173 participants. Individual interviews were conducted with school-based staff 
members (for example, school transfonnation facilitators, City Year program managers and 
team leaders, Communities In Schools site coordinators, instructional coaches, school adminis- 
trators, and school counselors) and district-based staff members (for example, Diplomas Now 
instructional facilitators, field managers, school and student support services facilitators, 
Implementation Support Team representatives, and school district leaders). Focus groups were 
conducted with parents, students, teachers, and City Year corps members. (See Appendix Table 
C. 1 for the number and type of respondents by school.) 


Methods and Analysis 

Measuring Diplomas Now Fidelity of Implementation 

“Fidelity of implementation” focuses on the extent to which DN schools implemented 
the model inputs as intended, considering both the Diplomas Now model in its entirety and 
those aspects of it deemed most critical by the Implementation Support Team. Other than the 
Program Staff Training and Development input, the inputs align with the Four Pillars (Teacher 
Teams and Smaller Learning Communities, Curriculum and Instruction with Professional 
Development, Tiered Student Supports, and Can-Do Culture and Climate). Each of the nine 
inputs consists of a set of more specific components identified by the Implementation Support 
Team. Given the complexity of the Diplomas Now model and school-wide reform, a total of 
111 components were identified under the nine Diplomas Now inputs, based on their im- 
portance in the context of the logic model as assessed by the Implementation Support Team. For 

4 Corrin et al. (2014). 
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each component, the evaluation team and the Implementation Support Team designed a fidelity 
scale to determine adequate implementation fidelity. 5 The fidelity of implementation matrix 
(see Appendix Table B.2) contains the inputs, their components, the components’ operational 
definitions, and the components’ fidelity scales. 

As noted previously, the Implementation Support Team determined that a subset of the 
inputs and components were critical to successful implementation of the Diplomas Now 
model. 6 It defined critical inputs and components as those that DN schools would have to 
implement adequately (as defined by the fidelity scales) in order to successfully implement the 
Diplomas Now model as a whole. It identified six of the nine inputs as critical and 62 of the 1 1 1 
components (across all nine inputs) as critical. Table 3.1 lists all of the Diplomas Now inputs 
and identifies those considered critical. 7 As Table 3.1 shows, the nine inputs vary widely both in 
their number of components and in the number of components identified as critical. In addition, 
the complexity of individual components may also vary. Although an input with more compo- 
nents or more critical components may be more challenging to implement, it is likely that other 
factors also contribute to the ease or difficulty with which a school implements a given input. 
Explanations of some of these factors should emerge over the course of the evaluation. 

Two Metrics for Measurement of Fidelity 

Two metrics are used to measure fidelity of implementation: a continuous score and a 
categorical rating. 

1. The continuous score examines all nine inputs and 111 components and estimates 
how much or how ‘ "fully ” schools implemented the Diplomas Now model. 

2. The categorical rating focuses on the critical inputs and critical components, and 
assesses how well the DN schools implemented those aspects of the model consid- 
ered most important by the Implementation Support Team. 

The score and the rating provide complementary assessments of fidelity, one measuring 
the overall implementation of the Diplomas Now model, and the other measuring the imple- 
mentation of the specific aspects of the Diplomas Now model identified as most important by 
the Implementation Support Team. 


5 This is the first time the Diplomas Now partners have implemented an assessment of Diplomas Now 
program fidelity. As such, the validity of critical inputs and components have not been tested, nor have the 
thresholds for assessing the adequacy of implementation. Both may require future revisions. 

6 Although the Implementation Support Team identified critical inputs and components, and may have 
guided schools to focus on these items, the list of critical and noncritical components was not shared with 
Diplomas Now school-based staff members, teachers, or administrators. 

7 Appendix Table B.2 lists all components, including those deemed noncritical. 
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Diplomas Now 
Table 3.1 


Diplomas Now Inputs and Components Identified as Critical 


Model Inputs 

Critical 

Input 

Number of 
Number of Critical 

Components Components 

Pillar I. Teacher Teams and Small Learning Communities 

Strong Learning Environments 2 

Yes 

6 

5 

Pillar II. Curriculum and Instruction with Professional Development 

Curriculum for College Readiness 

No 

24 

4 

Professional Development and Peer Coaching 2 

Yes 

5 

2 

Pillar III. Tiered Student Supports 

T iered Intervention Model 2 

Yes 

3 

2 

Student Supports 2 

Yes 

24 

19 

Student Case Management 2 

Yes 

14 

5 

Pillar IV. Can-Do Culture and Climate 

Integrated On-Site Support 2 

Yes 

11 

9 

Family and Community Involvement 

No 

6 

1 

Program Staff Training and Development 

No 

18 

15 


SOURCE: Diplomas Now fidelity of implementation matrix 2013. 


NOTE: a Strong Learning Environments, Curriculum for College Readiness, and Professional 
Development and Peer Coaching include components specific to middle school or high school. 


Independently, each measure provides useful but different information; taken together, 
they provide flexibility in understanding implementation, allow for detailed discussion of 
fidelity, and help to shape the implementation story. For example, instances where the continu- 
ous score and categorical rating are consistent could indicate steady implementation across all 
aspects of the model; low scores and high ratings might indicate that schools made a priority of 
critical aspects of the model; and high scores and low ratings might suggest that many aspects 
of the model were implemented, but that schools had difficulty in implementing some of the 
more critical aspects. 

The continuous score reflects all inputs and components of the Diplomas Now model. 
A score ranging from 0 to 1 is calculated for each input, based on its components. (Within each 
input the components are standardized, because fidelity scales and criteria for components 
vary.) The average of the scores for all inputs is a school’s continuous score. The continuous 
score therefore represents the proportion of Diplomas Now components implemented at a 
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school, and provides flexibility in that it credits schools for partially implemented components. 
The continuous scores for all schools are then averaged to produce an overall continuous score. 

To illustrate fidelity calculations, Table 3.2 presents an excerpt from the fidelity matrix, 
highlighting the input Strong Learning Environments and its high school components, the scales 
and criteria for assessing adequate fidelity, and sample responses from a hypothetical school. 8 
One can calculate a continuous score for Strong Learning Environments using Table 3.2. First, 
the components are standardized to a 0-to-l scale (for example, a component scaled from 0 to 2 
is recoded so that 0 = 0, 1 = 0.5, and 2 = 1) and then averaged together. In this example, there 
are five components of the Strong Learning Environments input: small learning communities, 
interdisciplinary teams, Diplomas Now site-based meetings, Diplomas Now site-based collabo- 
ration, and 4x4 block scheduling. The average of the fidelity scale responses for these compo- 
nents equals 0.46 (that is, 1 + 0.8 + 0 + 0.5 + 0, divided by 5). This is the school’s continuous 
score for Strong Learning Environments. Scores for the remaining inputs are calculated similar- 
ly, and the average of all input scores provides the school’s continuous score. The average of 
continuous scores from all schools provides an overall continuous score that estimates what 
proportion of the Diplomas Now model (from 0 to 1) was implemented across all DN schools 
during the first year. 

The categorical rating focuses on the implementation of critical inputs and critical 
components. The rating seeks to show how schools are implementing those components 
deemed most important to the model by the Implementation Support Team. Specifically, the 
categorical rating addresses six critical inputs and 62 critical components and assigns each DN 
school one of four ratings: 

1. Low: successful implementation of fewer than three critical inputs 

2. Moderate: successful implementation of at least three critical inputs 

3. Solid: successful implementation of at least five critical inputs 

4. High: successful implementation of eight or more inputs, including at least five crit- 
ical inputs 

In order to be considered successful on an individual input, model implementation at a 
school must meet the implementation adequacy thresholds for all of the components identified 
as critical. This is the case regardless of the school’s success with the other components of that 


8 Strong Learning Environments consists of six components: four that apply to middle and high schools, 
one that applies solely to middle schools, and one that applies solely to high schools. 
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Diplomas Now 
Table 3.2 


Example of a Diplomas Now Program Model Input, Related Components, and Fidelity Measures: 

Strong Learning Environments 


Component 

Operational Definition 

Fidelity Scale 11 

Criterion 
for Adequacy 

Sample Response for 
Hypothetical School 

Small learning 
communities 

Interdisciplinary teams of teachers who 
work with the same small group of students 

0: No 
1: Yes 

1 : Adequate 

1: Yes 

Interdisciplinary 
teacher team 
meetings 

Meetings where interdisciplinary core 
teachers discuss shared students 

0: Do not/rarely occur 

0.2: Occur monthly 

0.4: Occur biweekly 

0.6: Occur weekly 

0.8: Occur multiple times a week 

1 : Occur daily 

0.5: Adequate 

0.8: Occur multiple 
times a week 

Site-based team 
standards 

Site -based team (administrator, school 
transformation facilitator, project manager, 
and site coordinator) standards for 
collaboration, communication, and 
decision making 

0: No 

0.5: Partially/in process 
1: In place 

0.5: Adequate 

0.5: Partially/in 
progress 

Site-based team 
meetings 

Brief meetings for site-based team to 
review program implementation 
(approx. 30 minutes) 

0: Once a month or less 
0.5: Biweekly 

1: Weekly or more frequently 

0.5: Adequate 

0: Once a month 
or less 

4x4 block 
(high school only) 

4 class periods of 75 to 90 minutes that 
meet daily (or at least 4 days a week) 

0: No 

0.5: Hybrid/acceptable 

alternative 

1: Yes 

0.5: Adequate 

0: No 


(continued) 



Table 3.2 (continued) 


SOURCE: Diplomas Now fidelity of implementation matrix 2013. 

NOTES: This table only provides Strong Learning Environments components that apply to high schools. One middle school-specific component 
of Strong Learning Environments has been omitted. 

a Scales have been standardized to assist with the calculation of implementation metrics. 



input, regardless of the reason the component was not implemented (for example, district 
policies and procedures), and whether or not the component was partially implemented. 

As shown in Table 3.2, there are five components of Strong Learning Environments 
applicable to high schools. The Implementation Support Team identified four of these as critical 
to Diplomas Now implementation: small learning communities, interdisciplinary teams, 
Diplomas Now site-based meetings, and 4x4 block scheduling. 

Considering the example depicted in Table 3.2, this school met the criteria for adequate 
implementation for small learning communities (that is, teams of teachers work with the same 
small groups of students) and frequency of interdisciplinary team meetings (that is, occurring 
multiple times a week), but not for frequency of Diplomas Now site-based meetings (that is, 
biweekly) or 4x4 classroom blocks (that is, a hybrid schedule or acceptable alternative). Since 
implementation at the school did not achieve adequate fidelity to the model for all of this input’s 
critical components, this school would not be considered successful on the Strong Learning 
Environments input. Conducting a similar process on the remaining eight inputs would provide 
a final count of the number of inputs successfully implemented, leading to a categorical rating 
for each school. 


Overall Implementation Fidelity 

As noted in Chapter 2, baseline findings suggest that many schools were already familiar with 
or practicing reforms similar to those Diplomas Now offers, which may have positively or 
negatively affected implementation fidelity at some schools. For example, schools may have 
benefited if they already had structural components in place like daily schedules with four 
extended periods. Alternatively, schools may have encountered challenges in adopting new 
curricula that conflicted with their existing ones. The evaluation’s current data-collection plan 
does not allow for deeper analysis of these two possibilities. It is important to underscore, 
however, that the schools already implementing activities similar to Diplomas Now were not 
implementing the comprehensive Diplomas Now model, which depends on the coordination of 
practices, a systematic approach, and the identification and timely support of at-risk students. 

Table 3.3 provides summative first-year fidelity findings across all DN schools, sepa- 
rated by middle and high schools. First-year continuous score results indicate that DN schools 
implemented a majority of Diplomas Now components, with an overall average continuous 
score of 0.61 (ranging from 0.39 to 0.76 for individual schools, with a standard deviation of 
0.11). Turning to the model inputs considered most important by Diplomas Now leaders, 50 
percent of schools received a “low” categorical rating and 50 percent received a “moderate” 
rating. DN schools ranged from successfully implementing none of the critical inputs (one 
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Diplomas Now 
Table 3.3 


Summative First-Year Fidelity of Implementation, 
Findings Among Diplomas Now Schools 


Characteristic 

All DN 
Schools 

DN Middle 
Schools 

DN High 
Schools 

Fidelity of implementation continuous score 

Overall score 

0.61 

0.63 

0.59 

Fidelity of implementation categorical rating (%) 

Low implementation 

50.0 

52.9 

46.7 

Successful on no critical inputs 

3.1 

5.9 

0.0 

Successful on at least 1 critical input 

25.0 

17.6 

33.3 

Successful on at least 2 critical inputs 

21.9 

29.4 

13.3 

Moderate implementation 

50.0 

47.1 

53.3 

Successful on at least 3 critical inputs 

34.4 

29.4 

40.0 

Successful on at least 4 critical inputs 

15.6 

17.6 

13.3 

Solid implementation 

0.0 

0.0 

0.0 

Successful on at least 5 critical inputs 

0.0 

0.0 

0.0 

Sample size 

32 

17 

15 


SOURCES: Diplomas Now fidelity of implementation program staff surveys, spring 2012 and 2013. 

NOTE: For the categorical rating, schools are considered successful on a given input if they 
adequately implemented all of its components considered critical to the Diplomas Now model. 


school) to successfully implementing four of the six critical inputs (five schools). The continu- 
ous score of 0.61 suggests that, on average, model implementation had some traction during the 
first year. Still, the accompanying low and moderate categorical ratings show that even though 
on average the study schools were implementing a majority of the components, none of the 
schools were able to meet the standards set by the Implementation Support Team for “solid” 
implementation of the model because they did not adequately implement all of the components 
deemed critical to successful implementation. There were no significant differences between 
middle and high schools or between wave 1 and wave 2 schools. 
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Input-Level Implementation Fidelity 

Tables 3.4 and 3.5 categorize the nine model inputs by pillar, and provide continuous scores and 
categorical ratings for each input. Table 3.4 provides continuous scores that represent the 
average proportion of all components implemented for each input across all schools. These 
continuous scores suggest that DN schools had the most success implementing Tiered Student 
Supports (Pillar III) and the greatest difficulty implementing Curriculum and Instruction with 
Professional Development (Pillar II) in the first year of the program. The continuous scores also 
reveal that among the inputs, DN schools were most successful in implementing Integrated On- 
Site Support (0.82) and two Pillar III inputs, Tiered Intervention Model (0.76) and Student 
Supports (0.72). Taken together, this means that Diplomas Now was most successful at combin- 
ing data-driven identification of student needs (catching students who are off track) with the 
means to respond to those needs through coordinated programs and personnel (getting those 
students back on track). These are essential functions of the Diplomas Now focus on students’ 
progress toward high school graduation. 

In contrast, Curriculum for College Readiness (0.30) and Professional Development 
and Peer Coaching (0.44) under Pillar II were the most difficult to implement fully. Although 
there may be several reasons for these findings, it seems likely that Pillar III may have been 
implemented more successfully because its components complemented existing practices in 
schools. The influx of additional on-site staff members and persomiel made it easier to success- 
fully adopt a tiered intervention model that used new technology to assess student needs, and to 
include multiple student support services — such as tutoring and after-school programs — in 
the existing school structure. In contrast, Pillar II may have been more challenging to implement 
because its components may have required school staff members to change their existing 
practices. It may take more time for schools to adopt new curricula and peer coaching models 
because their effective implementation depends on the trust and investment of school adminis- 
trators and teachers. 

Table 3.5 presents the percentage of DN schools that adequately implemented all of the 
critical components of each input (that is, the percentage rated as successful for each input), as 
well as the percentage of critical components that schools not rated as successful did implement 
adequately. The percentage of critical components adequately implemented at unsuccessful 
schools indicates how close those schools came to a successful rating for each input. Table 3.5 
reveals that compared with other model inputs, higher percentages of DN schools successfully 
implemented all critical components of Tiered Intervention Supports (88 percent), Family and 
Community Involvement (66 percent), and Professional Development and Peer Coaching (56 
percent). These inputs also seem to represent an “all-or-nothing” scenario, as schools that failed 
to adequately implement all of the critical components also on average adequately implemented 


50 



Diplomas Now 
Table 3.4 


Input-Level First-Year Fidelity of Implementation Continuous Score, 
Findings Among Diplomas Now Schools 


Model Inputs 

Number of 
Components 

All DN Schools 
Average 
Continuous 
Score 3 

DN Middle Schools 
Average 
Continuous 
Score 3 

DN High Schools 
Average 
Continuous 
Score 3 

Pillar I. Teacher Teams and Small Learning Communities 

Strong Learning Environments' 3 

6 

0.67 

0.72 

0.61 

Pillar II. Curriculum and Instruction with Professional Development 

Curriculum for College Readiness 

24 

0.30 

0.34 

0.26 

Professional Development and Peer Coaching 1 ’ 

5 

0.44 

0.47 

0.40 

Pillar III. Tiered Student Supports 

Tiered Intervention Model 1 ’ 

3 

0.76 

0.75 

0.77 

Student Supports” 

24 

0.72 

0.77 

0.67 

Student Case Management 1 ’ 

14 

0.69 

0.68 

0.70 

Pillar IV. Can-Do Culture and Climate 

Integrated On-Site Support 1 ’ 

11 

0.82 

0.83 

0.80 

Family and Community Involvement 

6 

0.48 

0.53 

0.42 

Program Staff Training and Development 

18 

0.61 

0.58 

0.64 

Sample size 


32 

17 

15 


(continued) 



Table 3.4 (continued) 


SOURCES: Diplomas Now fidelity of implementation program staff surveys, spring 2012 and 2013. 

NOTES: a Each component was rescaled to a 0-1 metric; the continuous score is calculated as the average of the rescaled scores across all 
components within a given input. 

indicates model inputs designated as critical to the Diplomas Now model. 
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Diplomas Now 
Table 3.5 


Input-Level First-Year Fidelity of Implementation Categorical Rating, 
Findings Among Diplomas Now Schools 




All DN Schools 

DN Middle Schools 

DN High Schools 

Model Inputs 

Percentage 
of Schools 
Number of Successful 

Critical on All Critical 
Components Components 2 

Average 
Percentage 
of Critical 
Components 
Implemented 
Adequately at 
Unsuccessful 
Schools 15 

Percentage 
of Schools 
Successful 
on All Critical 
Components 2 

Average 
Percentage 
of Critical 
Components 
Implemented 
Adequately at 
Unsuccessful 
Schools 15 

Percentage 
of Schools 
Successful 
on All Critical 
Components 2 

Average 
Percentage 
of Critical 
Components 
Implemented 
Adequately at 
Unsuccessful 
Schools 15 

Pillar I. Teacher Teams and Small 
Learning Communities 

Strong Learning Environments 0 ' 

5 

0.0 

60.2 

0.0 

60.3 

0.0 

60.0 

Pillar II. Curriculum and Instruction 
with Professional Development 
Curriculum for College Readiness 0 

4 

40.6 

21.1 

35.3 

9.1 

46.7 

37.5 

Professional Development and Peer 
Coaching* 1 

2 

56.3 

10.7 

52.9 

12.5 

60.0 

8.3 

Pillar III. Tiered Student Supports 
Tiered Intervention Model* 1 

2 

87.5 

37.5 

88.2 

50.0 

86.7 

25.0 

Student Supports* 1 

19 

9.4 

74.0 

17.6 

78.6 

0.0 

69.8 

Student Case Management* 1 

5 

31.3 

69.1 

29.4 

65.0 

33.3 

74.0 

Pillar IV. Can-Do Culture and Climate 
Integrated On-Site Support* 1 

9 

50.0 

81.3 

47.1 

84.0 

53.3 

77.8 

Family and Community Involvement 

1 

65.6 

0.0 

70.6 

0.0 

60.0 

0.0 

Program Staff Training and Development 

15 

12.5 

79.0 

5.9 

77.9 

20.0 

80.6 

Sample size 


32 


17 


15 
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Table 3.5 (continued) 


SOURCES: Diplomas Now fidelity of implementation program staff surveys, spring 2012 and 2013. 

NOTES: a In calculating the categorical implementation rating, a school is considered "successful" for a given input if it adequately implements all of 
the critical components within that input. 

b The average percentage of critical components implemented adequately at unsuccessful schools excludes schools that were considered successful 
(that is, adequately implemented all critical components). 

c Although Strong Learning Environments includes five critical components, middle schools and high schools each have four critical components, as 
one component is solely for middle schools and one is solely for high schools. Similarly, Curriculum for College Readiness includes four critical 
components, of which two are solely for middle schools and two are solely for high schools. 
d Indicates model inputs designated as critical to the Diplomas Now model. 
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low percentages of the critical components (all less than 40 percent). DN schools struggled to 
implement all critical components of Strong Learning Environments (0 percent), Student 
Supports (9 percent), and Program Staff Training and Development (13 percent). However, on 
average schools that did not successfully implement all of the critical components of these 
inputs did adequately implement the majority of them (60, 74, and 79 percent of each input’s 
critical components, respectively). 

As Table 3.4 and Table 3.5 show, categorical ratings and continuous scores do not al- 
ways agree, providing two views of Diplomas Now implementation. For five of the nine inputs, 
the continuous scores indicate that on average the DN schools implemented at least two-thirds 
of the components (that is, their continuous scores were greater than 0.66). Yet for only one of 
these inputs are a high percentage of schools rated as successful in implementing the input’s 
critical components: the Tiered Intervention Model (successfully implemented in 88 percent of 
schools). For three of the other four inputs with high continuous scores — Strong Learning 
Environments, Student Supports, and Student Case Management — fewer than a third of the 
schools implemented all critical components; 50 percent of schools adequately implemented the 
critical components of the remaining input with high continuous scores, Integrated On-Site 
Support. Thus, although schools did implement many activities related to those inputs, they still 
have work to do in implementing all of those inputs’ critical components. For the three inputs 
with the lowest continuous scores — Curriculum for College Readiness (0.30), Professional 
Development and Peer Coaching (0.44), and Family and Community Involvement (0.48) — 
schools had moderate success implementing the critical components (41 percent, 56 percent, 
and 66 percent of schools, respectively, successfully implemented all critical components). 

Table 3.6 summarizes interview and focus-group data by listing the frequency with 
which each input was discussed related to four implementation themes. Program and school 
staff members were asked whether each input was easy to implement or challenging to imple- 
ment, whether they perceived it to have had an impact, and whether its implementation had 
yielded lessons for the Diplomas Now model in general and its continued implementation at 
their schools. Not surprisingly, given the fidelity scores and ratings, teachers expressed mixed 
feelings about the usefulness of the peer coaching, and most often raised the Professional 
Development and Peer Coaching input as one that should yield lessons for continuing imple- 
mentation. More focus-group and interview participants also noted difficulty in implementing 
the Diplomas Now curriculum than said it was implemented effectively; the two Pillar II inputs 
were the only ones more often discussed as challenging to implement than successfully imple- 
mented. In contrast, teachers attributed improvements in student course work to the Student 
Supports input, crediting among its other components the Early Warning Indicator meetings and 
the positive behavior support provided by City Year corps members. Similarly, respondents 
across all seven case- study schools most frequently discussed the Tiered Intervention Model, 
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Diplomas Now 
Table 3.6 


First-Year Interview and Focus-Group Findings, 

Number of Sessions in Which Diplomas Now Program Input Was Discussed, by Implementation Theme 




Implementation Themes 


Model Inputs 

Identified as 
Effectively 
Implemented 

Identified as 
Challenging 
to Implement 

Perceived Positive 
Impact on 
School or Students 

Lessons Learned 
for Future 
Implementation 

Pillar I. Teacher Teams and Small Learning Communities 

Strong Learning Environments' 1 

11 

5 

3 

5 

Pillar II. Curriculum and Instruction with Professional Development 

Curriculum for College Readiness 

8 

13 

2 

9 

Professional Development and Peer Coaching 1 ' 11 

11 

19 

1 

30 

Pillar III. Tiered Student Supports 

Tiered Intervention Model 1 

25 

17 

7 

15 

Student Supports 1 

33 

21 

12 

11 

Student Case Management 1 

16 

5 

2 

5 

Pillar IV. Can-Do Culture and Climate 

Integrated On-Site Support 1 

20 

12 

4 

10 

Family and Community Involvement 

12 

7 

9 

5 


(continued) 



Table 3.6 (continued) 


SOURCES: Diplomas Now program staff, school, and district interviews and focus groups, spring 2012 and 2013. 

NOTES: Data were collected at seven Diplomas Now schools representing four districts. Data-collection activities included 49 interviews and 28 
focus groups with a total of 173 participants (25 participants per school, on average). 

This table reflects the number of interviews and focus groups in which each theme was discussed. Each interview or focus group was only 
counted once regardless of the number of times the theme was discussed during the interview or focus-group session. 

The column labeled "identified as effectively implemented" refers to instances in which interview/focus-group respondents reported that a 
Diplomas Now program input either was necessary to the success of the Diplomas Now model or was executed effectively. The column labeled 
"identified as challenging to implement" refers to instances in which interview/focus-group respondents reported that a Diplomas Now program 
input was difficult to implement, did not work well, or was a barrier to the effective implementation of the Diplomas Now model. The column 
labeled "perceived positive impact on school or students" refers to instances in which interview/focus-group respondents described changes in 
school climate, student attendance, student behavior/discipline, or student course performance that they attributed to the implementation of a 
particular program input. The column labeled "lessons learned for future implementation" refers to instances in which interview/focus-group 
respondents talked about things they would have done differently, provided recommendations for other schools implementing the Diplomas Now 
model, described features unique to the first year of implementation, or described implications for sustaining Diplomas Now. 

indicates model inputs designated as critical to the Diplomas Now model. 

b While the inputs listed in the Diplomas Now logic model distinguish between "program staff professional development" and "professional 
development/peer coaching" for school staff members, the year-one case study analysis plan did not make this distinction and all types of training 
for both school and program staff members were assigned the same code. Future reports will make a distinction between training received by 
program staff members and training received by school staff members so that the analysis plan is more parallel to the inputs presented in the logic 
model. 



Integrated On-Site Support, and Student Supports, and usually in the context of effective rather 
than challenging implementation. 

The discussion of input-level implementation fidelity findings that follows is organized 
according to the Four Pillars of the Diplomas Now model. For each input, the discussion treats 
fidelity matrix data from program staff surveys as the core findings, supplemented where 
possible and pertinent with data from school staff surveys, interviews, and focus groups. 

Pillar I: Teacher Teams and Small Learning Communities 

The first pillar of Diplomas Now underscores the importance of clustering teachers and 
students into teacher teams and small learning communities that promote and generate student 
engagement and achievement. The Diplomas Now logic model input within this pillar is Strong 
Learning Environments. Table 3.7 presents additional detail regarding the Teacher Teams and 
Small Learning Communities pillar and its associated inputs and activities. Findings from this 
section include: 

• DN schools were most successful in establishing norms for collaboration, 
communication, and decision making among school administrators and Di- 
plomas Now program staff members, and in holding at least biweekly review 
meetings for the Diplomas Now program staff. 

• Strong Learning Environments fostered increased collaboration among 
teachers in both interdisciplinary and vertical subject-area teams, helping to 
prevent teachers from working in isolation. 

• Respondents viewed common planning time, when built into school sched- 
ules, as an essential practice that enabled teachers to communicate more ef- 
fectively with each other about student needs, reinforce and spread consistent 
messages across student cohorts, and help teachers hold students accountable 
for academic and behavioral requirements. 

Strong Learning Environments 

Under Strong Learning Environments, each DN school implements a staffing model 
that facilitates both interdisciplinary teacher teams and vertical subject-area professional 
learning communities with common planning time during the school day. Common planning 
time means that teachers meet at regularly scheduled times to discuss immediate student needs, 
plan interventions, and coordinate instructional expectations and practices. In a DN high school, 
the school building is organized into small learning communities or academies, each of which 
houses a moderate-sized group of students. Each small learning community/academy has 
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Table 3.7 

Pillar I: Teacher Teams and Small Learning Communities, 
Input and List of Component Types 


Input 

Component Type 

Component Definition 


Small learning communities 

Teams of teachers work with the same group 
of small students. 


Interdisciplinary teams 

Teachers representing different subject areas 
meet multiple times per week. 


Diplomas Now site-based 
meetings 

School-based program partners (Talent 
Development, City Year, and Communities In 
Schools) meet with school administrators. 

Small Learning 
Environments 

Diplomas Now site-based 
collaboration 

School-based program partners have a set of 
collaborative norms to guide their communication 
and decision making. 


School day includes 70- to 90-minute 
periods for all core academic classes 
(English/language arts, math, social studies, 
and science). 

4x4 block scheduling School day includes 4 70- to 95-minute class 

(high schools) periods that meet at least 4 days a week. 

NOTE: For more information on components see the frill fidelity matrix, Appendix Table B.2. 

dedicated administrators and counselors, and an interdisciplinary team of teachers — four to six 
teachers covering math, English/language arts, science, and social studies/history — who work 
with the same students throughout the year. Each high school has a ninth-grade academy and 
two or more thematic or career academies for grades 10 through 12. In addition, the daily school 
schedule is organized into extended class periods (four classes of 80 to 90 minutes each) to 
allow for in-depth instruction and differentiated instructional strategies that meet the needs of 
students with different learning styles. 

The continuous score indicates that schools implemented a moderate proportion of all 
components within Strong Learning Environments (0.67). Although no school adequately 
implemented all four components identified as critical to Strong Learning Environments, on 
average schools were successful in implementing the majority of critical components (60 
percent; see Table 3.5). 


Extended class periods 
(middle schools) 
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Program staff survey data indicate that DN schools were most successful in establishing 
norms for collaboration, communication, and decision making among school administrators and 
Diplomas Now program staff members (97 percent of schools successful), and in holding at 
least biweekly review meetings for the Diplomas Now program staff (78 percent of schools 
successful). Diplomas Now staff members who were interviewed emphasized that it was 
essential to secure the support of school leaders and teachers in order to set up well-functioning 
schedules for teachers and students working in small learning communities. Furthermore, 
Diplomas Now staff members reported that it was especially important to set a good tone and 
high expectations for students during the first year of implementation, and to establish rituals 
and traditions in each learning community. 

DN schools were also moderately successful at implementing small learning communi- 
ties, with 69 percent of schools organized so that each team of teachers worked with a common, 
small group of students. Eighty-seven percent of DN high schools used block scheduling or an 
acceptable alternative; block scheduling provides longer periods in which teachers can collabo- 
rate and hold meetings. 9 However, teacher surveys indicate that only 45 percent of core sixth- 
and ninth-grade teachers reported having adequate common planning time with an interdiscipli- 
nary team and only 52 percent reported having adequate common planning time with same- 
subject learning professionals. 10 Interview and focus-group data reveal that overall, Strong 
Learning Environments fostered increased collaboration among teachers in both interdiscipli- 
nary and vertical subject-area teams, which helped prevent teachers from working in isolation. 
In addition, when schools did implement common planning time school and program staff 
members saw it as essential, enabling teachers to communicate more effectively with each other 
about students’ needs, reinforce and spread consistent messages across student cohorts, and hold 
students accountable for academic and behavioral requirements across classrooms. 

The components of the Strong Learning Environment input reported to be most difficult 
to implement were scheduling interdisciplinary team meetings and creating extended class 
periods in middle schools. Based on program staff surveys, only 25 percent of schools reported 
scheduling interdisciplinary team meetings multiple times a week, and 53 percent of DN middle 
schools reported allowing 70- to 90-minute class periods for core academic courses. These 
findings are consistent with the data from school staff surveys, in which only 26 percent of 
administrators reported that interdisciplinary teams engaged in common planning more than 
once a week. Interviews and focus groups with Diplomas Now staff members suggest that some 
of the difficulty in implementing common planning time resulted from union requirements, 
schedule conflicts, and lack of support from school administrators. 


9 Rettig and Canady (1996). 

"'Adequacy for these components is defined as at least one to two hours of common planning time a week. 


60 



Pillar II. Curriculum and Instruction with Professional Development 

The second pillar of the Diplomas Now model promotes common instructional curricu- 
la and effective classroom practices. The Diplomas Now logic model inputs included in this 
pillar are Curriculum for College Readiness and Professional Development and Peer Coaching. 
Table 3.8 presents additional detail regarding the Curriculum and Instruction with Professional 
Development pillar and its associated inputs and activities. This section discusses these main 
findings: 

• Curriculum for College Readiness has the lowest continuous scores (0.30) of 
any input; only 4 1 percent of schools successfully implemented all of its crit- 
ical components, indicating that schools found it the most difficult to imple- 
ment. Very few DN middle schools offered reform curricula; in contrast, DN 
high schools had moderate success offering freshman seminars and creating 
separate ninth-grade academies. 

• DN schools generally succeeded in providing support to English/language 
arts and math teachers; however, instructional coaches struggled to meet the 
adequacy standards for frequency of planning, coteaching, and debriefing 
sessions with teachers. 

• Teachers expressed mixed opinions about the usefulness of peer coaching, 
while school administrators were more likely to express positive views about 
its effectiveness. 

Curriculum for College Readiness 

To prepare students for high school graduation and college, Diplomas Now provides 
schools with Talent Development Secondary’s reform curricula, which are intended to promote 
active learning, close skill gaps, develop mature thinking, and improve achievement. Diplomas 
Now’s curricula for middle schools integrate high school readiness and evidence-based core 
academic principles in math, English/language arts, science, and social studies. DN middle 
schools also have access to the Mastering the Middle Grades Curriculum , designed to facilitate 
the transition to middle school through instruction in life and study skills. For high schools, 
Diplomas Now offers a freshman seminar curriculum to ease the transition into high school, and 
evidence-based college preparatory curricula in core subjects. 

Of all the Diplomas Now model inputs, schools had the most difficulty fully imple- 
menting Curriculum for College Readiness, with a continuous score of 0.30 (see Table 3.4). 
Additionally, only 41 percent of schools successfully implemented all of its critical compo- 
nents (two critical components for middle schools and two for high schools). Middle schools 
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Table 3.8 


Pillar II: Curriculum and Instruction with Professional Development, 
Inputs and List of Component Types 


Input 

Component Type 

Component Definition 


Reform 

English/language arts 
and mathematics 
curricula 

Middle schools have implemented the Student Team Literature 
curriculum and an evidence-based math curriculum, or 
approved alternative interventions. 

High schools have implemented specific reform curricula for 
each grade level, such as Transitions to Advanced 
Mathematics and Reading and Writing in Your Career 
for freshmen. 

Curriculum 
for College 
Readiness 

Reading and 
mathematics labs 

Middle schools have implemented Savvy Readers' Lab or 
an approved alternative. 

Middle school and high schools have implemented the 
Computer and Team Assisted Mathematics Acceleration 
(CATAMA) program or, an approved alternative. 

Reform climate and 
success skills 
curriculum 
(middle schools) 

Middle schools have implemented the Mastering the Middle 
Grades curriculum and a program focused on building a 
positive school climate in all grade levels. 


Ninth-grade success 

academies 

(high schools) 

High schools are structured to include a separate 
academy for ninth-graders with its own administrators, 
teachers, and counselors. 


Freshman seminar 
(high schools) 

A seminar is offered in the first semester of high school to 
acclimate students to school culture and expectations. 

Professional 
Development 
and Peer 

Instructional 
coaching for 
English/language arts 
and math teachers 

English/language arts and math teachers are provided with 
weekly help from coaches (including planning support, 
coteaching, modeling, and debriefing). 

Coaching 

Freshman seminar 
support 

Freshman seminar teachers receive specific professional 
development and support. 


NOTE: For more information on components see the full fidelity matrix, Appendix Table B.2. 
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struggled to implement the reform curricula provided by Diplomas Now or adequate alterna- 
tives, with only 41 percent doing so for English/language arts and 29 percent for math. 11 
Additionally, very few DN middle schools (12 percent) offered the Mastering the Middle 
Grades Curriculum , which includes instruction in life skills and study skills. 

In contrast, program staff surveys reveal that DN high schools were moderately suc- 
cessful in offering freshman seminars (73 percent did so) and in creating separate academies for 
incoming ninth-grade students (60 percent). As shown in Table 3.4, the continuous score of DN 
high schools (0.26) was lower than that of DN middle schools (0.34). However, a higher 
percentage of high schools than middle schools had success adequately implementing all critical 
components (47 percent compared with 35 percent). The school staff surveys also highlighted 
this difference between middle and high schools: only 52 percent of middle school administra- 
tors reported instituting a course on effective transitions, compared with 86 percent of high 
school administrators. 

Students and teachers at case-study schools reported that the Diplomas Now curricula 
helped students transition into middle school or high school by teaching essential skills — like 
taking notes and delivering presentations — that could be applied in other classes. Some 
schools, however, did not implement the transition courses at all and others only implemented 
some elements of them. Teachers and administrators sometimes reported that the Diplomas 
Now transition courses “took up a lot of room” in student schedules, and schools often did not 
decide who would teach them until late in the summer, with the result that some teachers 
assigned to them did not receive adequate training. 

Professional Development and Peer Coaching 

At every DN school, instructional coaches are assigned to work with school administra- 
tors and with English/language arts and math teachers. Coaches provide initial training for all 
transition courses, instruction in how to teach an extended class period, professional develop- 
ment and in-classroom support, and customized training and workshops. In addition, DN 
schools are encouraged to identify and train local teachers to coteach, model lessons, serve as 
curriculum coaches, and support the full implementation of the Diplomas Now model. Diplo- 
mas Now instructional facilitators provide guidance to the instructional coaches at multiple DN 
schools, and also work directly with teachers in schools. 


"Diplomas Now reform EnglisManguage arts and math curricula include The Savvy’ Readers’ Lab and 
The Computer and Team Assisted Mathematics Acceleration Lab (CATAMA). Together they provide 
additional resources for students who are considerably behind in reading and math. Under the Diplomas Now 
model, schools can opt to implement existing curricula or programs already targeting these students. 
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The average continuous score for Professional Development and Peer Coaching is 0.44 
(see Table 3.4), and 56 percent of DN schools successfully implemented the input’s two critical 
components (see Table 3.5). Program staff surveys reveal that DN schools generally provided at 
least one period a week of support to English/language arts and math teachers (63 percent and 
59 percent, respectively), although instructional coaches struggled to provide biweekly plan- 
ning, coteaching, and debriefing sessions with teachers (41 percent for English/language arts 
and math). Not surprisingly in light of the Diplomas Now high school curriculum findings, 73 
percent of DN high schools provided professional development and job-embedded support 
directly geared to freshmen seminar teachers. The strong correlation between training and 
support for freshmen seminar teachers and success in offering these seminars in high schools 
underscores the importance of adequate professional development and training. 

During interviews and focus groups, teachers expressed mixed views regarding the use- 
fulness of peer coaching. Some teachers stated that they greatly benefited from peer coaching, 
while others saw it as an added burden. School administrators were more likely than teachers to 
express positive views about the effectiveness of peer coaching, but noted also that some 
teachers were not open to change or did not believe that they needed coaching. Instructional 
coaches and teachers from some schools indicated that the training held at the beginning of the 
school year was poorly organized, which negatively influenced the interest and support that 
teachers displayed for the Diplomas Now curricula and resulted in instructional coaches feeling 
ill prepared. 

Instructional coaches and instructional facilitators reportedly had the greatest impact 
when they spent time building relationships with teachers to leam more about their concerns, so 
that they could tailor their coaching sessions and training accordingly. 

Pillar III. Tiered Student Supports 

The third pillar of the Diplomas Now model relates to the provision of intensive and 
timely aid to students through a tiered system of interventions and resources wherein students 
with greater needs receive more intensive services. The Diplomas Now logic model inputs 
included in this pillar are Tiered Intervention Model, Student Supports, and Student Case 
Management. Table 3.9 presents additional detail regarding the Tiered Student Supports pillar 
and its associated inputs and activities. This section describes these main findings: 

• The majority of DN schools successfully integrated a tiered intervention 
model into the school day and implemented an Early Warning Indicator 
(EWI) system to alert teachers when students begin to demonstrate off-track 
indicators. 
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Table 3.9 


Pillar III: Tiered Student Supports, 
Inputs and List of Component Types 

Input Component Type Component Definition 


Tiered intervention A plan for integrating the use of the EWI system and 
model integrated in scheduling EWI meetings (see below) is in place. 
the school day 


Tiered Coordinated early 

Intervention warning indicator 
Model (EWI) system 


A data system is in place to alert teachers when students 
demonstrate tendencies that suggest they are moving 
off the graduation track. 



Collaborative, 
interdisciplinary 
EWI meetings 

Teams of teachers from different subject areas discuss 
and plan interventions for students shown to be 
at risk of not graduating according to the EWI system. 


Attendance and 
behavior assistance 

Students identified as at risk (Tier II) receive attendance and 
behavior coaching from City Year corps members. 


Academic assistance 

Students identified as at risk (Tier II) receive literacy and 
math tutoring from City Y ear corps members. 

Student 

Supports 

Classroom support 

City Y ear corps members offer additional support to students 
identified as at risk (Tier 11) during their English/language arts 
and math classes. 

After-school 

programs 

City Year corps members support after-school and extended 
learning time programs and recruit students identified as at 
risk (Tier II) for these programs. 


Whole-school 

activities 

City Year corps members help organize activities such as 
health fairs and career days. 

Student Case 

Students identified as most at risk (Tier III) are provided with 
Case management of individual student plans. The Communities In Schools site 
Tier 111 students coordinator manages attendance, behavioral, and academic 

interventions for them as needed. 

Management 

Whole-school 

services 

The Communities In Schools site coordinator provides 
college/career, enrichment/motivation, family-engagement, 
life-skills, and mental and physical health services, along with 
community service project opportunities. 


NOTE: For more information on components see the full fidelity matrix, Appendix Table B.2. 
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• EWI meetings facilitated infonnation sharing, collaboration, and improved 
working relationships between school staff members and program staff 
members, and some teachers attributed improvements in students’ course 
work to infonnation shared during EWI meetings. 

• Student Supports — such as tutoring and after-school programs — were of- 
ten identified as having a positive effect on student behavior and the overall 
school climate. DN schools were most effective in offering embedded sup- 
port for English/language arts and math classrooms (for example, individual 
student attention and small-group tutoring during class periods). However, 
many schools struggled to extend this resource to all classrooms or to im- 
plement after-school programs effectively. 

• All DN schools provided some case management to students identified as 
needing Tier III intervention, but many struggled to provide academic assis- 
tance to these same case-managed students beyond the services already pro- 
vided in Tier II. 

Tiered Intervention Model 

Each DN school establishes an early intervention system that uses EWI system data to 
trigger interventions targeted to student needs. EWI system data provide schools with on- and 
off-track indicators and early warning flags related to students’ attendance, behavior, and 
course performance — or as the Diplomas Now literature refers to them, the ABCs. The goal 
of the EWI system is to ensure that the right students receive the right intervention at the right 
time and at the right intensity. At each school, an EWI interdisciplinary team — which 
includes the Talent Development school transfonnation facilitator, administrators, teachers, 
City Year corps members, and Communities In Schools site coordinators — meets regularly to 
review and analyze EWI data. Through this analysis, the EWI team identifies interventions to 
meet the needs of all students (Tier I); targeted interventions (such as small-group activities or 
individual tutoring) for students who are not succeeding even after Tier I whole-school or 
classroom interventions (Tier II); and intensive interventions (for example, implementing a 
behavior-management system) for students at the greatest risk of falling off the path to gradua- 
tion (Tier III). 

The Tiered Intervention Model was by far the most successfully implemented input, 
with a continuous score of 0.76 and 88 percent of DN schools successfully meeting all critical 
components (see Tables 3.4 and 3.5). Identifying students in need (that is, students who are 
underperfonning in attendance, behavior, or course work) and then connecting them with staff 
members and programs that can address their needs is a crucial element of the Diplomas Now 
model intended to get students back on track toward graduation. As such, a strong start in the 
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implementation of tiered interventions shows promise that schools can provide continuing 
support to help students progress toward their diplomas. Although it can be challenging to get 
EWI systems in place and coordinate among three organizational partners the tiered programs 
meant to meet students’ needs, program staff surveys indicate that 75 percent of DN schools did 
integrate a tiered intervention model into the school day; 94 percent coordinated an EWI system 
to alert teachers when students begin to demonstrate off-track indicators (that is, measures of 
attendance, behavior, and course grades below established thresholds); and 91 percent sched- 
uled biweekly collaborative and interdisciplinary EWI meetings. Moreover, all administrators 
reported that their schools regularly tracked student-level perfonnance data in order to identify 
at-risk students. School staff survey data suggest that a subset of schools faced challenges with 
the frequency of EWI meetings: only 48 percent of administrators indicated that EWI meetings 
occurred at least weekly (although 8 1 percent reported that EWI meetings occurred more than 
once a month). 

Diplomas Now program staff members at high schools and middle schools frequently 
identified Tiered Intervention Model services as effectively implemented (Appendix Table C.2). 
Interviewed participants also noted the successes, challenges, and lessons they had learned from 
the first year of implementing the Tiered Intervention Model. For example, some teachers 
attributed improved student course work to information shared during EWI meetings. As one 
teacher said: 


They’ll bring students into our EWI meetings and show them, “Look, this is 
where you are in all your classes. All your teachers know where you’re at right 
now.” And when they see that and they know that all of their teachers know that 
they’re behind, that helps as well. So I’ve seen an improvement from the begin- 
ning of the year for some students with course perfonnance. 

EWI meetings were most often perceived as successful at those schools where the 
school transfonnation facilitator had built strong working relationships with teachers and 
administrators. These relationships were essential for ensuring that administrators held teachers 
accountable for attending EWI meetings, and that teachers were actively engaged in the EWI 
process. Interview and focus-group participants also noted that EWI meetings could serve as a 
way to encourage teachers to talk about students, that they facilitated infomiation sharing and 
collaboration, and that they improved working relationships between school staff members and 
program staff members. However, participants also noted that their demanding work schedules 
made it challenging for teachers to attend frequent EWI meetings. To alleviate this challenge, 
interviewees suggested that administrators and program staff members work with teachers to 
identify a meeting schedule convenient for all participants, and offer stipends to teachers for 
attending EWI meetings after school hours. 


67 



Student Supports 

Students identified by EWI data as requiring Tier II interventions receive additional 
support services from City Year corps members, which can include behavior and attendance 
coaching or management, after-school programs, academic tutoring (both individual and 
group), one-on-one mentoring, and near-peer role modeling. Students also receive in-class 
support services, such as one-on-one or small-group tutoring, from corps members embedded 
primarily in their English/language arts and math classes. 

Continuous scores for Student Supports are among the highest of all the inputs (0.72), 
although only 9 percent of DN schools adequately implemented all 19 critical components, (see 
Table 3.4 and Table 3.5). These findings are not surprising given the great number of compo- 
nents identified as critical. Moreover, the 91 percent of DN schools that did not adequately 
implement all critical components of Student Supports did succeed in implementing most of 
them (74 percent). 

Program staff survey data also reveal general success in the implementation of Student 
Supports. All DN schools offered additional year-round, in-class support structures for Eng- 
lish/language arts and math classrooms (100 percent) and provided at least four whole-school 
activities. An overwhelming majority (94 percent) also provided adequate behavior and attend- 
ance coaching to students. 12 These findings are consistent with those of the school staff surveys, 
in which administrators reported that students were offered weekly opportunities for academic 
tutoring (99 percent), behavior or anger management support (78 percent), and academic 
enrichment activities (93 percent). Moreover, the majority of sixth- and ninth-grade Eng- 
lish/language arts and math teachers (82 percent) reported that students participated in tutoring 
at least once a week. In contrast, survey responses from sixth- and ninth-grade core course 
teachers reveal one shortfall in Student Supports: only 38 percent reported that behavior 
coaching for frequently disruptive students was provided either often or as needed, suggesting a 
gap in services for at-risk students. 

Interestingly, although DN schools effectively offered additional in-class support 
structures, many struggled to provide that support to all classrooms. Only 50 percent of DN 
schools provided embedded City Year corps members in an adequate percentage of sixth- and 
ninth-grade English/language arts classrooms, and 41 percent did so in an adequate percent- 
age of math classrooms. 13 Schools also struggled to meet the Diplomas Now goals for after- 
school programs, with only 41 percent integrating corps members into existing after-school 

12 Adequacy for these components is defined as at least one student receiving behavior or attendance 
coaching per City Year corps member. 

13 Adequacy for this component is defined as providing embedded corps members in at least 75 percent of 
English/language arts and math classrooms. 
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programs, and only 47 percent providing after-school programs for at least three-quarters of 
the school year. 

Of all the inputs, Student Supports was most often identified during interviews and fo- 
cus groups as having had a positive impact on student behavior and school climate during the 
first year of implementation (Appendix Table C.2). One district administrator noted the benefits 
of Student Supports, highlighting positive changes to school culture: 

City Year has promoted things like positive behavior support. They provide af- 
ter-school tutoring, they make great relationships, great connections, and rela- 
tionships with our kids as mentors, as role models, and 1 think more than any- 
thing else, they are just an active, energetic, passionate group of young people 
that are committed to social change and their positive attitude helps change the 
culture of schools. 

Teachers most often expressed support for placing City Year corps members in their 
classrooms to work one-on-one with students and provide them individual attention. According 
to a middle school teacher: 

Because I’m a first-year teacher, 1 think having [corps members’] support in the 
classroom has helped me tremendously, especially with the struggling students ... 
since 1 don’t know how to be able to help every single student at one time, 
they’re there to help either pull out students or work with a group of students 
who need some extra support. 

The interviews and focus groups, however, revealed challenges during the first year of 
implementation that resulted from poorly defined expectations about the role corps members 
should play in classrooms. Some teachers saw corps members as a “distraction” and noted both 
advantages and disadvantages to the near-peer relationship. On the one hand the near-peer 
relationship facilitated trust between corps members and students, but in other cases corps 
members struggled to define their mentoring role. These findings from a selection of DN 
schools suggest that training for corps members should emphasize the importance of setting 
appropriate boundaries with students, and provide additional guidance on building relationships 
with teachers to ensure well-defined roles and expectations for corps members in the classroom. 

Student Case Management 

The Communities In Schools site coordinator provides case management to students 
whose indicators suggest that they are off track and require Tier III interventions, the most 
intensive support. The site coordinator conducts a needs assessment to determine the range, 
scale, and scope of the specialized support these students need. Next, the site coordinator 
develops an individual plan for each student, provides referrals for social services, coordinates 
individual or group counseling, completes home visits, and develops peer support groups. 
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Depending on the needs identified, case management can include academic assistance and 
career or college prep activities. Working daily or weekly with at-risk students, the site coordi- 
nator serves as a resource for these students and monitors their progress. Case management 
services are intended to reduce barriers to engagement and learning and lead to improved 
student outcomes. 

The continuous score of 0.69 for the Student Case Management input (see Table 3.4) 
suggests that schools implemented most of its components. Only 31 percent of DN schools 
successfully implemented all five critical components of Student Case Management, but on 
average the remaining 69 percent of schools successfully implemented 69 percent of the critical 
components (see Table 3.5). Compared with attendance and behavior interventions for students 
identified as needing them (75 and 91 percent of schools did an adequate job of implementing 
such interventions, respectively), schools had the most difficulty providing interventions to 
students identified as needing Tier III academic assistance (66 percent). 14 It is possible that, in 
light of the many opportunities for academic assistance offered by the City Year corps members 
as part of the Student Supports input (Tier II), site coordinators may have found it difficult to 
provide additional or alternative academic assistance. 

In general, schools were most successful in providing enrichment and motivational 
services (97 percent adequate), and in developing individual plans for case-managed students 
(91 percent adequate). 15 In addition, 88 percent of DN schools provided adequate needs-based 
resource interventions to case-managed students. 16 School and program staff members at 
case-study schools reported that site coordinators improved communication between the 
school and students’ homes, keeping students on track and parents better informed. Similarly, 
some students reported that site coordinators cared about their well-being, and administrators 
often noted that site coordinators provided additional support to their most at-risk students. As 
one high school principal said, “[Communities in Schools] has been a great model to give 
support to the kids who really need extra help beyond what [City Year] and [Talent Devel- 
opment] can offer.” 

However, some program and school staff members reported that they did not clearly 
understand how the site coordinator’s role differed from that of the school counselor. Similarly, 
at some case-study schools a single student could be seeing multiple counselors, making it 


14 Adequacy for these components is defined as providing appropriate interventions to at least 75 percent of 
Tier III case-managed students identified as needing them. 

15 Adequacy for this component is defined as providing individual student plans to at least 75 percent of 
case-managed students. 

16 Adequacy for this component is defined as providing appropriate resources and interventions to at least 
75 percent of case-managed students identified as having nonacademic needs (such as needs for food, clothing, 
or shelter). 
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difficult for the site coordinator to define a distinctive role. In other instances, site coordinators 
took on administrative duties that limited their availability to perform case-management 
activities. These findings highlight how important it is for site coordinators to work closely with 
school counselors to clarify the roles and responsibilities of each and to reduce redundancy. 
Improved coordination of services would enable site coordinators to provide more intensive 
continuing interventions to a smaller number of students, while school counselors serve a larger 
portion of the student population. 

Pillar IV. Can-Do Culture and Climate 

The fourth pillar of the Diplomas Now model is about developing a school-wide culture 
of success. Adding to the foundation of the previous three pillars, it emphasizes raising expecta- 
tions and providing engaging learning experiences. The Diplomas Now logic model inputs 
included in this pillar are Integrated On-Site Support and Family and Community Involvement. 
Table 3.10 presents additional detail regarding the Can-Do Culture and Climate pillar and its 
associated inputs and activities. This section discusses these main findings: 

• Integrated On-Site Support is among the most fully implemented of the Di- 
plomas Now model inputs, with most schools able to hire the full staff envi- 
sioned by the model. 

• The findings identified two important considerations when selecting on-site 
support staff members. First, involving school administrators in the selection 
process can help ensure that they view the staff members selected as “good 
fits” for the school’s needs and culture. Second, on-site support staff mem- 
bers with a history of working in a school district may understand that dis- 
trict’s culture better, making it more likely that teachers will believe in them, 
invest in them, and trust them. 

• The majority of DN schools engaged families and the community by offering 
workshops for parents, by ensuring that teachers made contact with parents at 
least once during the school year, and by scheduling parent-teacher confer- 
ences to include families and core teachers. Despite these efforts, many 
schools struggled to sustain active family and community engagement. 

Integrated On-Site Support 

To carry out all the programmatic, structural, and curricular components of the Diplo- 
mas Now program, numerous on-site staff members are needed. These staff members make it 
possible for schools to plan and implement the Diplomas Now model. 
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Diplomas Now 
Table 3.10 


Pillar IV: Can-Do Culture and Climate, 
Inputs and List of Component Types 


Input 

Component Type 

Component Definition 


School 

Transformation 

Facilitator 

The school transformation facilitator — who coordinates 
reform efforts including reform curricula, professional 
development, and the tiered intervention model — is in place 
prior to the first day of school. 

Integrated 

On-Site 

Instructional 

coaches 

Math and English/language arts coaches who provide 
professional development to teachers are in place at the school 
at least half the time. 

Support 

Communities In 
Schools site 
coordinator 

The Communities In Schools site coordinator — who manages 
the cases of Tier III students and provides additional whole-school 
services — is in place before the first student progress report. 


City Year corps 
members 

There is at least one City Year corps member for every 49 students. 
These volunteers provide attendance and behavior coaching, 
academic tutoring and in-class and after-school support. 

Family and 
Community 
Involvement 


A school action plan is in place to develop, implement, and 
evaluate work on family and community engagement. 

Engaging parents 

Parent volunteers are recruited and trained, parent-teacher 
conferences are scheduled with all parents, parent workshops 
are offered, and teachers send information to parents on how 
to help their children. 


NOTE: For more information on components see the full fidelity matrix, Appendix Table B.2. 


• A Talent Development school transformation facilitator works with the 
school’s leadership team to organize smaller learning communities or acad- 
emies within schools. Facilitators also work with leadership teams to analyze 
data, establish a positive school climate, and encourage effective leadership 
and teaching practices. 

• Instructional coaches provide instructional and curricular assistance to math 
and English/language arts teachers. They also collaborate with freshman 
seminar teams (interdisciplinary teams of teachers who work with the same 
ninth-grade students throughout the year) to develop lessons, projects, and 
other teaching strategies. 
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• A Communities In Schools site coordinator makes use of existing com- 
munity and school resources to organize whole-school interventions that are 
available to any student in the school. These activities can include health 
screenings, career and college fairs, donations of school supplies, and moti- 
vational events. The coordinator also provides case management to Tier III 
students as described above. 

• Ten to 20 trained City Year corps members are placed in each DN school 
at least four days a week. They are at the school before it opens and remain 
until the end of the after-school program, throughout the academic year. 

Corps members provide academic assistance inside and outside of class, en- 
gage with parents, run after-school programs, and operate programs intended 
to improve the school climate and attendance school-wide and for targeted 
groups of students. They also help teachers manage the data needed to oper- 
ate the EWI system. Corps members are overseen by a City Year program 
manager. 

With a continuous score of 0.82, Integrated On-Site Support was the most fully imple- 
mented Diplomas Now model input (see Table 3.4). Although only 50 percent of DN schools 
implemented all of this input’s nine critical components adequately (see Table 3.5), the other 50 
percent did implement the majority (81 percent) of the critical components. This widespread 
success is not surprising, given that this input is a precursor to many of the others. Numerous 
components — Early Warning Indicators, instructional coaching, student tutoring, and after- 
school activities — could not be accomplished without the additional on-site persomiel brought 
in by Diplomas Now. 

Additionally, most schools reported having all of the support staff members listed 
above, although their start dates and time at the school were often not quite ideal. Program staff 
surveys indicate that 8 1 percent of schools had a school transfonnation facilitator in place prior 
to the start of the school year. Eighty-one percent of schools had an English/language arts 
instructional coach at least half time, and 75 percent of schools had a math coach. Ninety-seven 
percent of all schools had a Communities In Schools site coordinator in place, with 8 1 percent 
starting prior to the first day of school. Seventy-eight percent of schools maintained an adequate 
corps-member-to-student ratio. 17 Additionally, 96 percent of administrators and 67 percent of 
sixth- and ninth-grade English/language arts and math teachers reported that City Year corps 
members worked with students more than once a week. 


17 An adequate corps-member-to-student-ratio is defined as 1 to 49 or fewer. 
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According to interview and focus-group data, school and program staff members re- 
ported high levels of both successes and challenges associated with the Integrated On-Site 
Support input (Appendix Table C.2). Participants in interviews and focus groups — both 
Diplomas Now staff participants and school staff participants — discussed successes more often 
than challenges. The successes they discussed related to their ability to get capable Diplomas 
Now support staff members in place at their schools. The challenges they discussed usually 
related to difficulties in developing the collaborative relationships between Diplomas Now staff 
members and school staff members that are important to model implementation. For example, 
some on-site support staff members reported that they struggled with their limited decision- 
making authority and influence at schools, which affected capable staff members’ ability to 
execute their assigned tasks. One school-based Diplomas Now staff member said: 

We have two phenomenal coaches here, and though the quality of instruction, in 
my opinion, is still low, it is because the coaches do not have the administrative 
power to really intervene, they can just coach and make recommendations but 
they can’t be the administrator that says, “Why aren’t you doing this?” 

Participants also often reported that on-site support staff members had more impact on 
newer teachers and less investment from more experienced teachers, who tended to be less 
receptive to “new” ways of doing things (for example, new curricula and peer coaching). 
Teachers at the case-study schools also varied in how they described their perceptions of the 
collaborations between the teaching staff and the Diplomas Now staff. Some teachers did not 
think on-site support staff members had an impact on their teaching, while others saw great 
value in these staff members. Chapter 4 discusses in further detail the nature of a few different 
types of staff collaborations, including collaborations between Diplomas Now and school staff 
members. 

Family and Community Involvement 

The Diplomas Now model assumes a concerted effort to engage in family and commu- 
nity partnerships that promote student success. As schools develop career academies, they 
involve local businesses, professionals, and government agencies in planning and implementing 
academy activities and work experiences for students, such as job shadowing and internships. 
City Year corps members lead community service opportunities for students, conduct outreach 
to parents, and in cooperation with the school’s staff and the site coordinator, organize and 
participate in school events like health fairs for family and community members. Through these 
efforts, DN schools seek to increase parent and community support and engagement in the 
schools and in student academic perfomiance. 

Overall, 66 percent of DN schools adequately implemented the single critical compo- 
nent under Family and Community Involvement, which assessed how well actual practices to 
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engage parents and community members lived up to their ideal versions (see Table 3.5). The 
average continuous score across all six components is 0.48 (see Table 3.4). Program staff 
survey data indicate that 88 percent of DN schools offered at least two to three workshops for 
parents, teachers in 78 percent of schools made contact with parents at least once during the 
school year to discuss how parents could help their children, and 75 percent of schools sched- 
uled parent-teacher conferences with families and core teachers. However, only 56 percent of 
schools identified a community engagement action team, which the model defines as a team of 
school, family, and community members established and trained to develop, implement, and 
evaluate the school’s work on family and community engagement. Only 19 percent formally 
recruited and trained parents to work as school volunteers. Additionally, although 96 percent of 
sixth- and ninth-grade teachers surveyed reported offering opportunities to parents to participate 
in school initiatives more than once a year, only 3 1 percent reported that parents or guardians 
actually participated. 

Program Staff Training and Development 

Although not included in any specific pillar, Program Staff Training and Development 
is an important input that influenced first-year implementation, and it was among the inputs 
most often discussed during the interviews and focus groups. Interviewees and focus-group 
participants most often viewed the training and professional development provided to program 
staff members as a vital process that helped to foster good working relationships and to orient 
program staff members to the different program components and the interactions among them. 
Because the Diplomas Now model has so many components, both school and program staff 
members emphasized the importance of defining the roles of all partners early and of spending 
time with school staff members devising strategies to integrate Diplomas Now’s components 
into the school. This section discusses these main findings: 

• The majority of DN schools did not successfully implement all 15 critical 
components of Program Staff Training and Development. However, on aver- 
age schools needed to implement only three to four more components to be 
rated as successful. 

• The most difficult components for schools to implement included having in- 
structional coaches attend English/language arts and math summer institutes, 
having site coordinators complete certification modules, and arranging con- 
tinuing EWI training for City Year corps members. 

Overall, the continuous score for Program Staff Training and Development is 0.61 (see 
Table 3.4). Although only 13 percent of schools successfully implemented all 15 critical 
components, on average the remaining DN schools did successfully implement the majority of 
them (79 percent, see Table 3.5). Program staff survey data indicate that 78 percent of schools 
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reported that all local Diplomas Now partners attended the three-day Summer Institute training 
session or received alternate training; that 81 percent of schools partially implemented a 
professional development plan to train teachers and school-based staff members on-site in 
curriculum, school climate, and working in teams; and that 94 percent of corps members had 
participated in relevant teacher professional development activities at least once or twice. 
Additionally, 59 percent of schools scheduled joint Diplomas Now staff, school administrator, 
and teacher planning sessions before the start of the school year. 

In the program staff surveys, three components of Program Staff Training and Devel- 
opment were reported as being the most difficult to implement: having instructional coaches 
attend summer institutes in English/language arts (47 percent) and math (44 percent), having 
site coordinators complete certification modules (41 percent), and arranging for City Year corps 
members to receive continuing training (34 percent). 

Similarly, interviewees and focus-group participants noted challenges related to pro- 
gram staff training and professional development during the first year of implementation. 
Program staff members at some schools did not leam about summer training sessions because 
they received no communication about them. In other cases, program staff members were hired 
after the training sessions occurred. In interviews, City Year corps members and teachers alike 
suggested scheduling joint training sessions that included teachers and Diplomas Now program 
staff members before the school year began, so as to more clearly define roles, responsibilities, 
and expectations. As one middle school teacher said: “I feel like we were given an idea of 
[corps members’] role, but not specifically how to incorporate them most effectively into our 
classroom.” 

Some corps members indicated during focus groups that most of their training had fo- 
cused on academic intervention, and suggested that more time be spent on managing the 
behavioral and emotional difficulties that students identify. Other corps members suggested 
additional training on developing lesson plans and dealing with students who have endured 
trauma, and better examples of situations that warrant mandated reporting. 

These insights highlight the importance of effectively coordinating summer training 
sessions and communicating with teachers and program staff members about them, to ensure 
that Diplomas Now school-based staff members are well prepared and properly trained when 
the school year begins. 


Summary 

Across DN schools, the Tiered Intervention Model, Integrated On-Site Support, and Student 
Supports were among the most fully implemented program inputs during the first year. Some 
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schools had activities associated with these inputs in place already, such as tiered support for 
students and common planning time for teachers. In some cases, the implementation of one 
input was a precursor to others. For example, many members of the Integrated On-Site Support 
staff needed to be in place before program inputs such as Student Supports and Student Case 
Management became possible. Implementing the full Diplomas Now model depends on Talent 
Development Secondary, City Year, and Communities In Schools’ ability to work together 
within a single school. It is not a simple undertaking for three different organizations that are 
new to a school to begin collaborating in a coordinated way that allows them to deliver services 
that address the goals of both their individual organizations and the Diplomas Now model as a 
whole. Thus it is not trivial that in their first year schools achieved relative success in imple- 
menting those aspects of the model closely linked to this collaboration, and the evaluation will 
investigate the potential value of this accomplishment over time. 

Curriculum development and peer coaching were among the most difficult inputs to 
implement. Qualitative data reveal that school staff members often were not convinced of the 
value of implementing new curricula, and some teachers were reluctant to be coached. These 
findings speak to the importance of involving a school’s staff in selecting program staff mem- 
bers, if possible, to encourage greater rapport and trust between school and Diplomas Now staff 
members. This is especially the case for instructional coaches, who frequently interact with 
teachers on a one-on-one, peer-to-peer basis. Also, it is possible that the tight timeline for 
bringing many of the DN schools into the project left less time than might be desirable for the 
Diplomas Now partners to build some of these relationships with school staff members before 
model implementation began. 

The importance of clarity regarding roles and responsibilities was also a recurring 
theme. Considering the large number of program components that had to be implemented at DN 
schools, and the number of additional staff members enlisted to support implementation, it is 
understandable that both Diplomas Now program staff members and school staff members were 
uncertain about each other’s roles and expectations. Especially during the first year of imple- 
mentation, it is essential to clarify the roles, responsibilities, and expectations of all parties by 
providing additional guidance, by building relationships between Diplomas Now staff members 
and teachers, and by offering continual program staff training, school staff training, and joint 
training. 


Although categorical ratings only ranged from low to moderate, a deeper analysis of the 
percentage of critical components implemented within each input reveals specific areas where 
targeted efforts could improve fidelity ratings substantially. For example, ensuring that instruc- 
tional coaches attend the summer institute, that site coordinators complete certification modules, 
and that continuing EWI training is available for City Year corps members would greatly 
improve the categorical rating for Program Staff Training and Development. Moreover, 
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ensuring that Diplomas Now program staff members are identified and complete their training 
before the school year begins would go far to improve ratings across multiple inputs. The 
Integrated On-Site Support input is critically important to implementation, because it is a 
precursor to many of the other Diplomas Now model inputs and components. In this way, 
concentrating on some of the most difficult-to-implement components could substantially 
strengthen fidelity of implementation in DN schools. 

Overall, program staff survey data indicate that Diplomas Now partners adequately col- 
laborated with each other, with the overwhelming majority of school-based staff members 
indicating they had access to each other for timely assistance. Chapter 4 continues to explore 
collaboration among Diplomas Now staff members and between Diplomas Now and school 
staff members during the first year of implementation, drawing primarily on qualitative case- 
study data. 
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Chapter 4 


Collaborative Interactions Among Partners 
in the Implementation of the Diplomas Now Model 


School improvement necessitates that people act, react, and interact in new ways, and effective 
collaboration is essential to such change. Lessons from the first year of implementation of the 
Diplomas Now model reflect many of the findings from the school refonn literature concerning 
interactions between external agencies and schools . 1 Lessons from Diplomas Now may be 
particularly illuminating in reflecting the experiences of staff members from three different 
organizations coming together to work with schools on a complex refonn model. Although 
these initial lessons are limited, they may be instructive to others engaging in similarly complex 
interactions with schools. 

The main points to take away from this chapter are: 

• Assigning Diplomas Now school-based staff members work spaces in close 
proximity to one another facilitates interaction conducive to implementation: 
continual communication, resource sharing, trust building and mutual aware- 
ness, and collective problem solving. 

• Clarifying and establishing roles and responsibilities early promotes cohesion 
within the Diplomas Now school-based staff and accountability among staff 
members for those responsibilities. 

• Hiring and situating Diplomas Now school-based staff members in the 
school before the academic year begins appears to facilitate cooperation that 
results in quick implementation of short-term program goals as well as long- 
er-tenn planning for collaborative initiatives. 

• Training sessions and forums that clearly communicate what Diplomas Now 
entails for a school and its personnel, and that include examples of successful 
outcomes, may help attract the interest and commitment of administrators 
and teachers. 

• Diplomas Now school-based staff members can anticipate that resistance 
from administrators and teachers may occur, especially in the beginning. 


Berends. Kirby, Naftel, and McKelvey (2001); Lachman and Wlodarczyk (2011); Rowan, Correnti, Mil- 
ler, and Camburn (2009); Smylie and Evans (2006); Spillane, Gomez, and Mesler (2009). 
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Continual efforts to build rapport — including flexibility and respect for the 
school’s context and culture — could mitigate those challenges. 

• City Year corps members appear to be the aspect of the Diplomas Now pro- 
gram most visible to students, suggesting that school-based staff members 
should consider how they can capitalize on the role of corps members to bet- 
ter understand student needs and respond to them. 

The Diplomas Now model requires that many individuals in different roles work to- 
gether over time to deliver whole-school support (Tier I), student support (Tier II), and student 
case management (Tier III). Effective collaboration is the heart of this complex school reform, 
which deploys staff members from Talent Development Secondary, City Year, and Communi- 
ties In Schools to interact with each other, school personnel, students, parents, and community 
members over the course of implementation. The program model asserts that individuals from 
the three partner organizations will establish shared norms for working together, communicate 
continuously, share leadership responsibilities, and make decisions together in order to achieve 
the program’s goals of improving student attendance, behavior, and course performance. 

The findings presented in this chapter are drawn primarily from qualitative data collect- 
ed through interviews and focus groups at seven Diplomas Now case-study schools in four 
school districts. Participants shared their perceptions in response both to specific questions 
about collaboration and to more general questions about factors that expedited or hindered 
program implementation. The chapter presents common findings across participants and 
schools, as well as individual perspectives and experiences. The findings are based on inter- 
views and focus groups with individuals who volunteered to participate. The research team 
applied systematic analysis consistent with best practices in qualitative research. In addition, 
data from the Diplomas Now program staff surveys indicate how partner organizations support- 
ed each other and the extent to which school and district personnel responded to the needs 
expressed by Diplomas Now school-based staff members. Given the breadth of the data that 
emerged, this chapter draws attention to themes that were commonly mentioned by participants, 
notes exceptions where relevant, and provides details to illustrate the perceptions of individual 
participants. However, it is not possible or practical to capture all the concepts that emerged 
from all of the 77 interview and focus-group sessions. 

Table 4.1 details the different types of collaborative interactions according to the fol- 
lowing implementation themes: facilitators of implementation and examples of effective 
collaboration; barriers to implementation and examples of ineffective collaboration; perceived 
positive impacts on school climate, staff, or students; and lessons learned for future implementa- 
tion. The interviewees and focus groups primarily discussed collaborative interactions 
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Diplomas Now 
Table 4.1 


First-Year Interview and Focus-Group Findings, 

Number of Sessions in Which Collaboration Among Diplomas Now Stakeholders Was Discussed, 

by Implementation Theme 

Implementation Themes 
Barriers to 

Facilitators of Implementation Perceived Lessons 

Implementation and Examples Positive Impact Learned 


Collaborative Interactions 

and Examples 
Collaboration 

of Ineffective School Climate, 
Collaboration Staff or Students 

for Future 
Implementation 

Among Diplomas Now school-based staff members 

20 

12 

3 

23 

Between Diplomas Now school-based staff and school administrators 

26 

15 

2 

19 

Between Diplomas Now school-based staff and teachers 

29 

26 

4 

24 

Between Diplomas Now school-based staff and students 

3 

2 

1 

4 

Between Diplomas Now school-based staff and other Diplomas Now staff members 3 

7 

8 

0 

4 

Between Diplomas Now school-based staff and school district staff members 

1 

2 

0 

4 


(continued) 



Table 4.1 (continued) 


SOURCES: Diplomas Now program staff, school, and district interviews and focus groups, spring 2012 and 2013. 

NOTES: Data were collected at seven schools representing four districts. Data-collection activities included 49 interviews and 28 focus groups with a 
total of 173 participants (25 participants per school, on average). 

This table reflects the number of interviews and focus groups in which each theme was discussed. Each interview or focus group was only counted 
once regardless of the number of times the theme was discussed during the interview or focus group session. 

The column labeled "facilitators of implementation and examples of effective collaboration" refers to instances in which interview/focus-group 
respondents reported that a Diplomas Now program input facilitated collaboration, which in turn contributed to the successful implementation of the 
Diplomas Now model, or provided examples of collaboration that was perceived as effective. The column labeled "barriers to implementation and 
examples of ineffective collaboration" refers to instances in which interview/focus-group respondents reported that a Diplomas Now program input 
hindered collaboration, which in turn led to challenges in the implementation of the Diplomas Now model, or examples of collaboration that was 
perceived as ineffective. The column labeled "perceived positive impact on school or students" refers to instances in which interview/focus-group 
respondents described changes in school climate, student attendance, student behavior/discipline, or student course performance that they attributed to 
the implementation of a particular program input. The column labeled "lessons learned for future implementation" refers to instances in which 
interview/focus-group respondents talked about things they would have done differently, provided recommendations for other schools implementing 
the Diplomas Now model, described features unique to the first year of implementation, or described implications for sustaining Diplomas Now. 

a "Other Diplomas Now staff members" includes regional and national staff members from Diplomas Now or the individual organizations of City 
Year, Communities In Schools, and Talent Development (for example, instructional facilitators, school and student support services facilitators, field 
managers, and managing directors). 



(1) among Diplomas Now school-based staff members, (2) between Diplomas Now school- 
based staff members and school administrators, and (3) between Diplomas Now school-based 
staff members and teachers. This chapter explores these three types of collaborative interactions 
in detail in the next three sections, followed by a fourth section on other types of collaborative 
interactions less frequently described by participants. As can be seen in Table 4.1, interviewees 
and focus-group members more often described these three types of collaborations as facilita- 
tors or examples of effective implementation than as implementation challenges. In addition, in 
reflecting on the first year of implementation, research participants frequently discussed 
“lessons learned” about how to strengthen these collaborations as implementation progresses. 

Each section briefly describes important features of the interactions, followed by a dis- 
cussion of the factors that facilitated or hindered relationships, including examples of successes, 
challenges, and lessons learned about working together to implement the Diplomas Now model. 
Finally, the chapter closes with a summary of contextual factors that influenced program 
implementation at the seven case-study schools. 


Collaboration Among Diplomas Now School-Based 
Staff Members 

As discussed in previous chapters, Diplomas Now school-based staff members include the 
Talent Development school transformation facilitator and instructional coaches; the City Year 
program manager, team leaders, and corps members; and the Communities In Schools site 
coordinator. They work together to facilitate whole-school reform by delivering Integrated On- 
Site Support and other services related to the Diplomas Now model. The school transfomiation 
facilitator guides the School Leadership Team and analyzes student data. Instructional coaches 
provide teacher professional development, such as training sessions and in-class guidance. 
School-based staff members from City Year include a program manager, who oversees City 
Year programs at the school; City Year corps members, who provide students with Tier II 
interventions and school-wide programs; and City Year team leaders, experienced corps 
members who assist the program manager and other corps members through peer leadership. 
The Communities In Schools site coordinator brokers community resources and provides 
student case management, often focusing on Tier III support. Each of the three partner organiza- 
tions plays a role in managing Diplomas Now reform at each school, and the Diplomas Now 
School Leadership Team consists of the school transfomiation facilitator, the program manager, 
and the site coordinator. 

The qualitative data indicate that Diplomas Now school-based staff members generally 
worked together to plan and accomplish shared goals. The Diplomas Now program staff survey 
data from 32 schools also reflect moderate to high levels of collaboration among Diplomas Now 
partners. Survey respondents generally reported that they could rely on each other whenever 
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they had a concern or issue. The survey asked Diplomas Now school-based staff members to 
rate the responsiveness of each Diplomas Now partner to concerns or issues, using a six-point 
scale (where 1 = “not at all,” 2 = “a little,” 3 = “to some extent,” 4 = “quite a bit,” 5 = “very,” 
and 6 = “completely”). Most Diplomas Now school-based staff members (ranging from 75 
percent to 94 percent across schools) reported that they could rely on the other partner organiza- 
tions “quite a bit,” “very,” or “completely.” These findings are encouraging given that the staff 
members from these three distinct organizations need to work together to support the implemen- 
tation of multiple components of a complex whole-school reform model. 

Qualitative data reveal that Diplomas Now school-based staff members worked togeth- 
er both formally and informally to plan for and accomplish shared goals. Fonnal interactions 
consisted of regularly scheduled meetings with agendas; informal interactions included continu- 
ing communications through impromptu discussions (sometimes text messaging) and un- 
planned check-ins (such as might occur when passing by each other’s offices). Planning 
activities usually fell into the following categories: 

• Preparing for events and activities (often involving City Year corps members and 
the Diplomas Now School Leadership Team) 

• Developing instruction (for example, instructional coaches working with corps 
members) 

• Devising strategies to meet long-term goals (often within the Diplomas Now 
School Leadership Team, sometimes involving the team leader) 

Participants offered many examples to illustrate how Diplomas Now school-based staff 
members worked together. For example, the school transformation facilitator’s responsibilities 
include providing student data to the Diplomas Now school-based staff, coordinating and 
leading Early Warning Indicator (EWI) meetings, and being, as one program manager stated, 
the “engine behind our collaboration.” One school transformation facilitator noted that the 
program manager, site coordinator, and school transformation facilitator need to clearly deline- 
ate roles and ensure that leadership responsibilities are distributed effectively across the school- 
based team. This leadership function appeared essential to successful Diplomas Now imple- 
mentation, given the complex interactions of multiple players. Although their collaborative role 
is not always clearly defined, site coordinators interact with other Diplomas Now school-based 
staff members by connecting students with services after EWI meetings, planning school 
programs, and serving as a primary point of contact when City Year corps members identify 
individual student needs. 

The program manager often serves as the link between the Diplomas Now School 
Leadership Team and corps members by setting expectations for corps member responsibilities; 
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at many schools, the team leaders operate as extensions of the program manager in helping 
corps members understand their responsibilities. With the program manager focused primarily 
on leadership responsibilities within the school and the local area, the team leaders can help 
manage corps members on a frequent, often day-to-day, basis. Corps members provide “people 
power” and serve as a bridge between the staff and students, giving the Diplomas Now School 
Leadership Team and instructional coaches insight about the issues students face and serving as 
an important source of encouragement and accountability for students. 

Four primary themes emerged from the qualitative data concerning interactions among 
Diplomas Now school-based staff members: 

• Building trust and rapport within the Diplomas Now team may foster an ef- 
fective partnership among the three partners. 

• Establishing and clarifying roles early will make it easier to hold Diplomas 
Now school-based staff members accountable for their responsibilities. 

• Assigning work spaces for Diplomas Now school-based staff members in 
close proximity to one another facilitates continual interaction. 

• By working together, Diplomas Now school-based staff members successful- 
ly fulfill program goals shared by the three organizations, as well as program 
goals specific to each individual organization. 

Participants often described collaboration among Diplomas Now school-based staff 
members as enthusiastic and positive. For example, one team leader observed, “We all work as 
a team and we all rely on each other for help.” Participants at one school indicated strong 
consensus about how well Diplomas Now school-based staff members, particularly the Diplo- 
mas Now School Leadership Team, worked together as an “effective partnership” and “support- 
ive team.” During an interview, one of the administrators from this school attributed the 
Diplomas Now school-based staffs cohesion to the clarity of their roles and responsibilities: 
“Everyone knew what the next person was doing, and so it kept things seamless.” The school’s 
fidelity data (implementation rating, moderate; implementation score, 0.76; number of critical 
inputs implemented adequately, four; and number of all inputs implemented adequately, six) 
confirm that implementation was progressing well — both overall and in comparison with other 
Diplomas Now schools. This implies that collaboration of this nature may be essential to the 
successful implementation of the Diplomas Now model. 

Other schools found it challenging to maintain intense collaboration throughout the 
year. For example, one program manager reported weekly meetings with the school transfor- 
mation facilitator at first, but said that as the year passed, their interaction dwindled to brief 
conversations passing in the hallways. Based on responses from several participants, it seems 
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important to “cement” relationships among Diplomas Now school-based staff members; some 
participants reported that team building and open communication helped to develop trust and 
cultivate collaboration. As a Diplomas Now field manager noted, “The first part is more like 
you’re feeling each other out.... Now we have the team working, and we’re very transparent in 
our meetings. So that built trust. So we have teamwork, transparency, and trust.” 

Many respondents indicated the need to establish the roles and clearly define the re- 
sponsibilities of Diplomas Now school-based staff members early in model implementation, 
thus creating norms for working together. Challenges arose in cases where Diplomas Now 
school-based staff members came on board after the school year began. There weren’t enough 
people available to implement all the model components early on, and those staff members who 
were in place faced difficulties planning program activities and services in advance, and in 
collectively establishing and branding the Diplomas Now model at the school. 

One school transformation facilitator observed, “I think starting off the year knowing 
that there wasn’t really a planning year, [the planning] sort of happened as the implementation 
was happening, and not all the partners began in September.” Early clarification of roles might 
address what the City Year corps members at one school articulated as their confusion about the 
responsibilities of the school transformation facilitator and the site coordinator. At a few 
schools, Diplomas Now school-based staff members reported that the site coordinator’s respon- 
sibilities were not well defined; in one case, the site coordinator described not being clear about 
what role to play. In another case, confusion about the site coordinator’s role arose because 
other people and organizations in the school delivered similar services; this underscored the 
need for Diplomas Now school-based staff members to communicate, coordinate, and integrate 
services with those other parties. Diplomas Now school-based staff members from two schools 
expressed frustration with other school-based staff members whom they perceived as not 
“pulling their own weight”; they emphasized the need for greater accountability and open 
channels to communicate such concerns. 

Securing office space in close proximity to one another facilitates collaboration among 
Diplomas Now school-based staff members because it expedites communication, resource 
sharing, awareness of one another’s work, and collective problem solving. Administrators and 
City Year corps members from one middle school and one high school noted that situating 
Diplomas Now school-based staff members close to one another created a central location to go 
for help. One administrator indicated that having the Diplomas Now school-based staff in the 
same location made it easy to go into that room and find someone to assist whenever it was 
needed. On the other hand, the program manager and site coordinator at another school shared 
an office, but the school transformation facilitator’s office was located in a separate section of 
the school building, a setup that reportedly presented challenges in executing the work — 
specifically, limited opportunity for informal communication. In cases where shared office 
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space is not an option, Diplomas Now school-based staff members may need to work more 
actively to ensure frequent communication and opportunities for collaboration. 

Although the three partner organizations share a common agenda of serving young 
people, planning together requires devising strategies to fulfill individual organizational goals as 
well as collective Diplomas Now program goals. Reported challenges included “different 
organizations ... working in the schoolhouse” that referred to things differently, demonstrating 
the need to develop a common language. Diplomas Now School Leadership Team members at 
some schools described how they worked together to focus on the overarching goal of student 
success. For example, a program manager stated, “It’s not about Communities In Schools 
looking good, or Talent Development looking good, it’s about our students receiving maximum 
benefits from our program.” Many participants suggested more emphasis on long-term plan- 
ning, which might streamline how Diplomas Now school-based staff members work together to 
achieve goals. 


Collaboration Between Diplomas Now School-Based 
Staff Members and School Administrators 

The Diplomas Now model seeks to supplement rather than supplant school personnel. Collabo- 
ration between Diplomas Now school-based staff members and school administrators is 
therefore critical to integrating the program into a school and sustaining school improvement 
over time. Administrators are key allies for successfully implementing Diplomas Now model 
components that may require modifying the school structure, such as extended periods and 
small learning communities. 

During interviews and focus groups, both school and Diplomas Now school-based staff 
members described the collaboration between the Diplomas Now School Leadership Team and 
school administrators, including the principal and assistant principals. A school transformation 
facilitator explicitly distinguished between “active” and “passive” support from administrators, 
a difference to which others also alluded. Active engagement occurs when administrators 
empower Diplomas Now school-based staff members by advocating for the program and 
serving as a resource for its implementation. One administrator reported this approach to 
involving the school transformation facilitator: 

If you have outside programs in your building, you cannot isolate them. It’s im- 
portant that you incoiporate them into the fabric of the building. And it was im- 
portant that I incoiporate Diplomas Now into my leadership team. Because if 
you’re going to have anything to do with curriculum, anything to do with my 
students, anything to do with my teachers, you need to be involved in a leader- 
ship process.... We work together hand in hand because the support I’m getting 
from her — I need her support and she needs mine. 
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Others described school administrators who gave passive support. These administrators 
informed Diplomas Now school-based staff members about district or school initiatives, 
allowed them autonomy, and approved initiatives, both formally through signatures and 
informally through “walk-and-talks.” As a school transfomiation facilitator said, “They gave us 
the green light to go ahead.... They gave us free rein.” Some participants preferred active 
support and others preferred passive support. The difference may depend on context. For 
example, in one school where teachers’ unwillingness to implement the Diplomas Now model 
presented a challenge, the school transfomiation facilitator expressed interest in more active 
involvement from school administrators. 

Consistent with the interview and focus-group findings, the Diplomas Now school- 
based staff surveys reflect a moderate to high degree of support from school administrators 
during the first year of implementation. The survey asked Diplomas Now school-based staff 
members to rate the responsiveness of school administrators to concerns or issues brought to 
their attention, using a six-point scale (where 1 = “not at all,” 2 = “a little,” 3 = “to some 
extent,” 4 = “quite a bit,” 5 = “very,” and 6 = “completely”). Talent Development Secondary 
and City Year staff members generally rated the responsiveness of school administrators in the 
middle to high range, with 75 percent of Talent Development Secondary and 72 percent of City 
Year staff members indicating that they could rely on school administrators “quite a bit,” 
“very,” or “completely.” 2 

Qualitative data related to the interactions between Diplomas Now school-based 
staff members and school administrators suggest several factors that expedite successful 
collaboration: 

• Maintaining the support of school administrators is better accomplished 
when Diplomas Now school-based staff members actively set realistic expec- 
tations for what the program will accomplish. 

• Administrator investment from the beginning of implementation is essential 
to the viability of the Diplomas Now model. 

• Diplomas Now school-based staff members benefit from being resilient dur- 
ing the initial challenges of implementation. 

• Opportunities for administrators to network with each other and share sto- 
ries related to successful reform may motivate them to engage in the pro- 
gram actively. 


2 Data from Communities In Schools staff members for this item were not available for this report. 



• Getting administrators to support the work of the Diplomas Now school- 
based staff may be a powerful influence making teachers more willing to 
take part in the program. 

It is important that school administrators not only understand the Diplomas Now model, 
but set reasonable expectations for the Diplomas Now school-based staff. Some administrators 
reported that Diplomas Now school-based staff members were very receptive to helping when 
necessary, and expressed appreciation for their willingness to do things “they aren’t supposed 
to.” Diplomas Now school-based staff members at the same school, however, recommended 
setting boundaries with administrators; this might prevent situations where administrators ask 
Diplomas Now school-based staff members to engage in tasks outside of their intended respon- 
sibilities, spreading them too thin. Several Diplomas Now school-based staff members stated 
that they “got sucked into” activities and responsibilities not directly related to their Diplomas 
Now responsibilities: conducting groups for the guidance counselor, providing social work 
interventions, and writing pink slips for poor student behavior. Depending on the school 
context, Diplomas Now school-based staff members may need to determine how to strike an 
appropriate balance between building rapport and setting expectations. 

Pillar IV — Can-Do Culture and Climate — specifies getting administrators to commit 
to a realistic yet optimistic vision about what the Diplomas Now program can achieve. Diplo- 
mas Now school-based staff members can play key roles in helping administrators gain that 
understanding. For example, one teacher confinned the observations that Diplomas Now 
school-based staff members made about their school leaders’ lack of understanding: “I don’t 
feel that the administration necessarily even knows what the program should look like from one 
class to the next.... I found that frustrating.” Similarly, one school transformation facilitator 
spoke about how important it is for school administrators to understand the commitment 
required to effect change: 

The leadership expected Diplomas Now to come in and save the day overnight, 
not understanding that it takes time. School transformation does not happen 
overnight, and for school transformation to happen you have to be willing to 
transform your thinking and your understanding. You have to be open to that and 
if you are not open to really changing the way that you think about students’ 
achievement ... then it is not going to transform. 

Not surprisingly, Diplomas Now school-based staff members said stakeholder invest- 
ment was important to successful program implementation. More telling, however, were the 
overwhelming number of responses from a broad range of participants (school district staff 
members, school administrators, and teachers) about the need for support from school adminis- 
trators. How administrators feel about the Diplomas Now model (whether they accept it, are 
neutral toward it, or reject it) is probably critical to implementation — as is how strongly they 
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feel about accepting or rejecting it — and Diplomas Now school-based staff members cannot 
assume that administrators will initially accept the program. For example, a school district staff 
member offered this recommendation: “You need principals buying [in] up front.... If not, they 
don’t take ownership and then it doesn’t work, no matter how well-intentioned all the players 
are.” Interview and focus-group participants at all seven schools described at least some success 
getting administrators on board with the Diplomas Now model. Administrators offered a few 
suggestions for how a Diplomas Now school-based staff can make its administrators more 
willing to engage in the Diplomas Now program actively: 

• Adapt to the needs of the school. 

• Understand the culture of the school. 

• Share information to ensure that everyone is on the same page. 

• Engage all the necessary players. 

Getting administrators to become actively engaged reportedly takes some time, and of- 
ten does not happen easily. At one school, for example, participants reported that the previous 
principal resisted the Diplomas Now model; the principal “wanted something else from the 
program” and had a “difference of opinion” about how implementation ought to occur. As this 
school’s transfonnation facilitator noted, “I think it’s a disservice to everybody if a principal 
truly doesn’t want this and it’s pushed on them ... because it’s not going to happen the way it’s 
supposed to.” In this case, a change in leadership presented new opportunities for the Diplomas 
Now school-based staff to draw attention to what the Diplomas Now model could offer the 
school. Many respondents from various schools described early resistance from administrators 
but also noted that, in time, most administrators became more enthusiastic, after seeing the 
changes in school climate, student behavior, and attendance that resulted from the program. 
According to one program manager: 

Administration was sure to point out that they were skeptical at first, but they are 
very pleasantly surprised at the impact that we had this year, in our first year. So 
I think we are in a really good place going into our second year, having the buy- 
in from administration and being empowered to do the work that we do ... taking 
it to the next level I think for next year, which is pretty exciting. 

Many participants discussed the value of school administrators visiting another Diplo- 
mas Now school or meeting principals from other schools implementing the Diplomas Now 
model. As one site coordinator noted, these visits allow a principal to see firsthand the potential 
impact of the Diplomas Now program, which is likely to increase the principal’s interest in its 
implementation at his or her school. Respondents from another school noted that taking part in 
the Diplomas Now Summer Institute enabled the Diplomas Now School Leadership Team and 
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school administrators to share the same vision coming into the school year. Additionally, one 
school administrator advised other school leaders to be open to “having that collaborative spirit 
where you don’t mind having those folks come in to talk and give suggestions,” a sentiment that 
might mitigate some of the tensions with administrators that Diplomas Now school-based staff 
members experienced at other schools. 

Collaboration between administrators and Diplomas Now school-based staff members 
also appears to be a stepping stone to engaging teachers with the Diplomas Now program. For 
example, one school administrator commented about the principal’s importance: 

The willingness for the school, the administrator to really work with [Diplomas 
Now] ... it was definitely the administrator who — she drives it. And not only 
that, she invited the staff to welcome them, the staff, many of them have wel- 
comed them to our school and looked to them for that support.... She set the tone. 

It becomes especially important for school leaders to be actively engaged in the Diplo- 
mas Now program when teacher resistance occurs, as exemplified in a scenario that one 
participant reported: teachers refused to attend EWI meetings and administrators did not require 
them to. 


Collaboration Between Diplomas Now School-Based 
Staff Members and Teachers 

As is true in many school reform models, in Diplomas Now teachers not only receive program 
services (instructional coaching, for example), but are also active and crucial implementers of 
key elements — delivering Curriculum for College Readiness, for example, and working with 
City Year corps members to provide in-class services to students. Schools took different 
approaches to the interaction between Diplomas Now school-based staff members and teachers. 
One program manager noted that the school transformation facilitator was the lead partner in 
collaborating with teachers at that school, while at other schools the entire Diplomas Now 
School Leadership Team exercised this responsibility. Interactions between instructional 
coaches and teachers also varied. Teachers in one focus group reported that coaching began on 
a daily basis and then shifted to weekly “random pop-ins.” One school’s model of weekly 
preclass meetings, observations, and postclass meetings was described by one teacher as “too 
intensive.” In contrast, the instructional coaches at another school also had teaching responsi- 
bilities that limited their opportunities to engage in this type of coaching cycle. 

To facilitate collaboration between Diplomas Now school-based staff members and 
teachers, participants recommended four approaches: 
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• Conduct training and discussion sessions to clarify what is expected of teach- 
ers and what resources are available. 

• Clarify the role of City Year corps members and set appropriate expectations 
for their classroom activities. 

• Continually build rapport with teachers, recognizing that some may initially 
oppose the Diplomas Now model. 

• Publicize small successes that demonstrate how the Diplomas Now program 
can create desired change. 

Across schools, participants reported challenges in fostering teacher understanding of 
the Diplomas Now model but also offered a range of strategies for doing so. Diplomas Now 
school-based staff members, teachers, and administrators all recommended more extensive 
training for teachers in the Diplomas Now model, including: 

• An organizational chart to illustrate “who’s who” in the Diplomas Now model 

• Discussion forums to help teachers clarify their understanding of the program 

• A summer retreat to introduce the Diplomas Now model to new teachers and serve 
as a refresher course for experienced ones 

• An initial training session to explain roles and responsibilities, with a follow-up ses- 
sion a month later to note successes and adjustments needed 

• Small group sessions to share infonnation 

• Assistance with classroom setup at the beginning of the school year to provide op- 
portunities for one-on-one conversations about the Diplomas Now model 

Many participants raised specific concerns about the clarity of the City Year corps 
member role, and some suggested ways to manage relationships between teachers and corps 
members. Although one school transformation facilitator reported having clearly defined the 
role of corps members to teachers in the beginning of the year, corps members at that school 
suggested that it would have helped if school administrators also communicated that infor- 
mation to teachers. One teacher described planning with a corps member to be certain that 
nobody “overstepped bounds.” A corps member stated the importance of sensitivity in respect- 
ing a teacher’s space to “avoid animosity.” Navigating these often fluid relationships, according 
to one team leader, requires that corps members figure out how to work with teachers. Corps 
members expressed appreciation for other Diplomas Now school-based staff members who 
corrected teachers when they referred to corps members as “kids.” During interviews and focus 
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groups, however, teachers and administrators used similar terms to refer to corps members, 
confirming that some do not view corps members as mature and responsible staff members of 
Diplomas Now. 

In some cases, the interactions between corps members and teachers changed over time. 
For example, one corps member observed that, “It was kind of interesting for that transition to 
happen, because at one point you didn’t want me to do anything, and now you want me to do 
everything.” Similarly, corps members at another school articulated the intent to scale back their 
initially extensive involvement in student behavior management. As one commented: 

So this year we have done things like reviewing pink slips, calling homes, send- 
ing students home, things like that, and we won’t do that because that’s not our 
job. And we have been kind of sucked into that role because of lack of just staff 
members to do that. We won’t do that next year. 

Although Student Supports — including behavior management — is a Diplomas Now 
model input, corps members are not intended to function as a school’s primary disciplinarians, 
as apparently occurred in this case. Another respondent noted the importance of providing 
concrete guidance to corps members who lack training in instructional methods, especially as 
they acquire more teaching responsibilities over the course of the school year. One corps 
member described using an infonnation sheet on teacher engagement to start initial conversa- 
tions that defined both parties’ expectations from the beginning. Based on the qualitative data, it 
seems important that the Diplomas Now School Leadership Team equip corps members with 
the tools they need to advocate for themselves, and simultaneously work with administrators to 
explain the role of corps members to teachers. 

Interview and focus-group participants across schools reported that it was difficult to 
get teachers invested in Diplomas Now, and that that lack of investment was a barrier to 
program implementation. For example, when asked which aspect of the program was most 
challenging to implement, a school transformation facilitator said that it was getting teachers on 
board at the beginning of the year. A school administrator confirmed this view, stating that it 
took several weeks to persuade teachers of the Diplomas Now program’s importance. In other 
cases, this resistance increased over time, as reflected in an instructional coach’s story about 
eventually interacting with a teacher “just for the show of working with him, but by the end of 
the year he completely resisted.” Superficial compliance may have a negative effect on the 
quality of Diplomas Now program implementation, and on schools’ ability to sustain the 
program. 

The qualitative data suggest that teachers took a long time to adjust to the Diplomas 
Now model components in their schools and classrooms, including assistance from corps 
members, instructional coaching, and EWI meetings. Corps members noted that teachers, 
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especially the more traditional and experienced of them, “have their own agendas and their own 
ideas, and they don’t want to try anything new, [and this] conflicts with changing the school 
culture.” Some teachers are not open to coaching because, as one teacher stated, “They think 
they are perfect even though there is always room to improve.” 

Another area of tension at several schools was the unwillingness of teachers to attend 
EWI meetings, as described in Chapter 3’s discussion of the Tiered Intervention Model. It 
was challenging to convince teachers to participate in additional activities, like EWI meetings, 
when they perceived their workloads as already intensive. As one program manager de- 
scribed, persuading teachers requires “sophisticated relationship building to win them over 
and make sure they are dedicated to coming each and every time.” Reinforcement from 
administrators was also noted as an effective strategy to ensure teachers attended these 
meetings. On the other hand, administrators at two schools reported that the implementation 
of the Diplomas Now program became an impetus to establish teacher team meetings and a 
formal, regularly scheduled opportunity to discuss student progress, work together, and 
connect with Diplomas Now school-based staff members. Other respondents noted that EWI 
meetings provided opportunities for teachers to talk about students and get on the same page 
— a model one participant recommended be expanded to other grades. Overall, the findings 
indicate that although some teachers are less amenable than others to intervention from 
Diplomas Now school-based staff members, effective communication about the benefits of 
the model can persuade many to accept the program. 

Diplomas Now school-based staff members at one school reported that because teach- 
ers were interested in the program from the outset, both school and Diplomas Now school- 
based staff members accomplished more than planned. At this school, Diplomas Now school- 
based staff members created relationships with teachers at the beginning of the year by meeting 
with them to hash out questions, concerns, and problems. Respondents from other schools 
described strategies to gamer teacher interest in the program, including intensive efforts to win 
teachers over. For example, one program manager reported that the Diplomas Now school- 
based staff secured the commitment of initially skeptical teachers by having “to bend over 
backwards a lot at the beginning of the year.” A few participants noted that effective leadership 
and rapport building persuaded teachers that Diplomas Now could become a real resource for 
improvement rather than just “another thing on their plates.” As one school transformation 
facilitator explained: 

If [teachers] don’t believe in it and see the value in it and feel supported in it, 
then you don’t even have a shot at getting it off the ground. So I really wanted to 
make sure that I was supporting teachers in the beginning and making sure that I 
was a resource for them. 
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Diplomas Now staff members at one school staged events such as honorary breakfasts 
to show their appreciation for teachers, and deployed the Diplomas Now program logo and 
banners to encourage more familiarity with the program inside the school. One instructional 
coach summarized what other participants described as the value of building rapport: “Relation- 
ships are everything. If you don’t have those relationships, then you will meet lots of resistance, 
depending on the teachers and their comfort level.” 

Drawing attention to small successes can raise teachers’ interest in implementing the 
Diplomas Now model. For teachers to sustain their ownership in the program, they need to see 
results. According to one school transformation facilitator: 

They need to know that this is teacher-based and teacher-driven work ... that we 
are outside partners coming into a school, but at the end of the day, we’re help- 
ing to organize the school in a way that gives them more leadership. 

One team leader told the story of a previously resistant teacher who heard about the im- 
pact of the program at the EWI meetings, and subsequently became much more open to 
working with corps members. Teachers reported directly that corps members had helped them 
reach struggling students with whom they could not connect. One school transformation 
facilitator confirmed that: 

Once [teachers] saw that our work was valid, and once we proved our levels of 
proficiency, and that what we do really matters, and it counts, and it’s very help- 
ful not only to the students but to them — they were willing to do whatever it 
took to assist us and help us out with making certain things happen. 

These findings about collaboration between Diplomas Now school-based staff mem- 
bers and teachers may benefit others. Knowing what challenges they might encounter, Diplo- 
mas Now school-based staff members can plan to lessen the concerns that teachers are likely to 
have, and patiently execute strategies to achieve their support over time. 


Other Findings About Collaborative Dynamics 

Although most of the qualitative data concerned the collaborative interactions already discussed 
in this chapter, to a lesser degree participants did reflect on other collaborative dynamics. This 
section describes interactions between Diplomas Now school-based staff members and students, 
between school-based staff members and other Diplomas Now staff members, and between 
school-based staff members and school district staff members. It also describes some contextual 
factors that influence collaborative dynamics. 

Diplomas Now school-based staff members engage with students by working together 
to deliver Integrated On-Site Support through a tiered intervention model. However, respond- 
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ents indicated that it is primarily City Year corps members who have frequent, direct interac- 
tions with students. Their interactions include individual student support services and Tier II 
interventions (for example, tutoring, in-class support, and behavioral management), as well as 
whole-school support services (for example, school-wide activities and attendance calls to 
parents). Across all seven schools, students offered in-depth accounts of their interactions with 
corps members when asked about the Diplomas Now program. In other words, corps members 
were the most visible aspect of the program to students. Students reported that: “They [corps 
members] go farther than what we do in class.... If you have a question on it, they’ll talk to you 
about it and try to explain it,” and “They sit there and just listen.... They’ll give you time and 
they’ll let you cry in a little comer in their classroom.” A school administrator added that: 

[Corps members] have several, quite a few, gatherings in the auditorium. They 
were able to cheer on the students, talk about their success, commend them for 
their hard work, and identify several young people that were exceptional. 

One site coordinator offered this example of one-on-one interactions between Diplomas 
Now school-based staff members and students: 

[Students] share a lot of infomiation with us as to what’s going on around them 
— classroom issues, teacher issues, home issues, whatever is going on they 
come to me, and they’re able to open up and share with me a lot. 

The interviews and focus groups indicate that Diplomas Now school-based staff mem- 
bers should consider the ways they can capitalize on the role of corps members to understand 
student needs and respond to them. 

Collaboration between the Diplomas Now School Leadership Team and other Diplo- 
mas Now staff members — the Diplomas Now Implementation Support Team, Diplomas Now 
Local Executive Team, and Diplomas Now school-based staff members from other schools — 
promotes program implementation through helpful discussions and resource sharing. For 
example, two school transfonnation facilitators described their school and student support 
services facilitator, who is part of the Diplomas Now Local Executive Team, as helpful and 
supportive. One school transfonnation facilitator observed: 

He comes in and he is always hands-on, big on teaming.... He has always given 
us our best practices for those types of things, whether it is from readings or 
whether it’s from things at some of his other schools. 

Program managers at two schools indicated that collaboration with other program man- 
agers in their district was helpful because it provided the opportunity to compare the lessons 
they had learned and the practices that had been effective for them. One of them, however, 
indicated that the regional meetings where this collaboration occurred also presented challenges 
because they meant time away from the school. Instructional coaches at two schools mentioned 
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collaborating with other coaches in their school districts. Across all four districts, participants 
spoke of the important role played by instructional facilitators (part of the Diplomas Now local 
support system for multiple implementing schools within a district) in implementing the 
program, because of their detailed understanding of the model. Instructional facilitators’ 
responsibilities included coaching the school-level instructional coaches and, in cases where 
instructional coaches were not well received by teachers, coaching the teachers directly. One 
school transformation facilitator asserted that the instructional facilitators, who visited twice a 
month, “are a huge help. At times now it almost gets to where it’s a little overused, but at the 
beginning of the year it was a huge resource that we definitely utilized.” On the other hand, a 
school administrator from another school wanted more frequent visits, stating that, “Instruction- 
al facilitator support is really critical.” Based on the findings from the qualitative data, effective 
assistance from local Diplomas Now staff members not based in the school was at times a 
helpful supplement to the Diplomas Now school-based team. 

Qualitative data indicate that Diplomas Now school-based staff members obtained the 
backing of district leaders in three out of the four school districts. However, Diplomas Now 
school-based staff surveys from the 32 Diplomas Now schools reflect lower levels of perceived 
district support. The survey asked Diplomas Now staff members to rate the responsiveness of 
their school districts using a six-point scale (where 1 = “not at all,” 2 = “a little,” 3 = “to some 
extent,” 4 = “quite a bit,” 5 = “very,” and 6 = “completely”). Talent Development Secondary 
and City Year staff members rated school district responsiveness in the lower range, with 44 
percent of Talent Development Secondary staff members and 53 percent of City Year staff 
members reporting that they could rely on their school districts to respond to their concerns 
“quite a bit,” “very,” or “completely.” 3 

When district leaders are interested in the Diplomas Now model, they may take op- 
portunities to share success stories with other school leaders. For example, one school 
transformation facilitator described how at a district meeting a district leader discussed the 
success the school had had with report card conferences. A district staff member commented 
that the district could improve its communication with school administrators about the 
Diplomas Now model: 

One thing we could do better as a district in partnership with Diplomas Now is 
training up front to clearly outline what is the role of Communities In Schools, 
what is the role of Talent Development Secondary, and what is the role of City 
Year in improving the conditions for your kids. 

One of the roles of a Diplomas Now Local Executive Team is to help ensure that the 
work of school-based Diplomas Now staff members is responsive to the district context and 

3 Data from Communities In Schools staff members for this item were not available for this report. 
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aligned with district goals. The Diplomas Now field manager from one school described efforts 
“to make sure that not only are we implementing the Diplomas Now model, but we are, if you 
will, vertically aligned to the district vision and goals.” The Local Executive Team can also 
advocate with district administrators in support of school-level refomis that are part of the 
Diplomas Now model. For example, coaxing from the Local Executive Team might help 
change the mind of a school district staff member who is not interested in adopting the Diplo- 
mas Now curriculum. 

Contextual factors, such as staff stability and resources, are significant because they re- 
late to collaboration and overall implementation. Schools experiencing staff turnover and 
layoffs, whether recent or anticipated, had difficulty engaging teachers in the Diplomas Now 
program. Replacement of school administrators could be an asset or a liability, depending on the 
support that the former and current leaders gave to the Diplomas Now program. As one school 
administrator commented: 

[The school] had some difficulties previously, and those areas were addressed; 
there’s a new administration in place as well as a number of new teachers, and 
everyone is really on board and working toward changing the dynamics of the 
school. 

District-driven transitions, such as pending school closures, budget cuts, and restructur- 
ing, may make teachers and administrators preoccupied with uncertainty and less vested in 
school reforms such as the Diplomas Now program. One teacher weighed in about this topic: 

With budget cuts, and not being able to have as many resources, I mean, 1 know 
that’s on everybody’s mind right now. So it is not what can be improved, but 
what are we going to do without? Flow are we going to keep even the same 
standards we have now with no money and no teachers and no support? 

In schools with particularly negative climates (because of either low teacher morale or 
poor student behavior or both), Diplomas Now school-based staff members faced extraordinary 
difficulties in promoting school improvement. 

Although the Diplomas Now model brings an influx of individuals into schools, re- 
spondents asked for other resources as well, such as data-system access, transportation to and 
from Diplomas Now program events, and designated space for after-school activities. Some of 
these challenges are out of the hands of the Diplomas Now school-based staff, as one site 
coordinator explained, “Transportation is a huge issue.... That’s something simple that we really 
don’t have any control over, but another big hurdle for us.” Although some contextual factors 
are within the influence of the Diplomas Now model, Diplomas Now school-based staff 
members would benefit from considering the other contextual factors that they cannot influence, 
and the implications of those factors for program implementation. 
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Summary 

The findings presented throughout this chapter suggest that the collaborative dynamics related 
to the Diplomas Now model are both complex to navigate and critical to the successful imple- 
mentation of the program. Two aspects of collaboration appeared to be most important across 
schools and various role groups: investment and role clarity. Administrators and teachers are 
key stakeholders whose engagement in implementing the Diplomas Now model inputs is 
essential. In order to become actively engaged, it is important that they understand the model 
through activities such as information sessions or meeting school staff members at other 
Diplomas Now schools. It is vital that Diplomas Now school-based staff members establish 
their purposes and roles, with teachers and administrators as well as among themselves; it is 
particularly helpful if administrators set a good tone and reinforce those expectations with 
teachers. It is challenging to integrate the goals of three partner organizations with those of a 
school and a district, and it can only be achieved if the main players work together toward their 
common purpose: improved individual student outcomes and school climate. Continual com- 
munication, including regular formal meetings and informal check-ins, helps build the trust and 
acceptance necessary for this collaboration. 

Diplomas Now is not unique in the challenges it faces as an external group coming into 
a school building, but the multiple partner organizations involved ensure that these challenges 
are considerable. That is, while the Diplomas Now reform model seeks strength from a partner- 
ship of organizations that provide complementary programs to schools, it also faces the chal- 
lenge of increasing the amount of collaboration that needs to happen to get the model in place. 
Lessons from the first year of implementation, especially those regarding investment and role 
clarity, are consistent with previous research on school turnaround efforts that rely on many 
individuals working together in the name of student success . 4 Effective collaboration is at the 
core of most, if not all, school reform. Other organizations engaged in similarly complex 
interactions with schools may benefit from the findings discussed in this chapter, such as 
insights about cultivating the active engagement of key players in schools, rolling out the 
program, coordinating services, setting clear expectations, and communicating effectively. 

Although this round of data analysis provided many insights about the launch of the 
Diplomas Now model, practical suggestions for how to address some of these challenges will 
probably emerge in subsequent years. The continuing implementation research will examine 
how Diplomas Now stakeholders have worked to overcome initial difficulties with implementa- 
tion, and consider how to sustain program efforts over time. 


4 Berends, Kirby, Naftel. and McKelvey (2001); Lachman and Wlodarczyk (2011); Rowan, Correnti, Mil- 
ler, and Camburn (2009); Smylie and Evans (2006); Spillane, Gomez, and Mesler (2009). 
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Conclusion 

Over the course of the 201 1-2012 and 2012-2013 school years, 32 secondary schools across 1 1 
school districts took on the challenge of implementing the Diplomas Now whole-school reform 
model. Designed to reduce the number of students who drop out and provide students with a 
college- and career-preparatory education, the Diplomas Now model brings multiple interven- 
tions to a school in a tiered, data-driven format supported by a team of staff members from the 
three Diplomas Now partner organizations: Talent Development Secondary, City Year, and 
Communities In Schools. At the same time, 62 secondary schools across the same 1 1 school 
districts (including the 32 schools implementing Diplomas Now) have begun participating in a 
national evaluation of the implementation and impact of Diplomas Now. This report is the first 
of three planned as part of this evaluation. It has introduced readers to the Diplomas Now 
partner organizations and the Diplomas Now school reform model. It has also provided initial 
background on the nature of the evaluation and described the sample of participating schools. 
The 62 participating schools serve students from low-income communities where there are 
challenges both within and outside of school that put many students’ progress to high school 
graduation at risk. These schools represent the kinds of schools in which the Diplomas Now 
organizations seek to work, and for which the model was designed. The schools were randomly 
assigned to implement the Diplomas Now model (DN schools) or to continue with or pursue 
other school improvement strategies (non-DN schools). Initial analyses confirm that randomiza- 
tion has resulted, as expected, in two comparable groups of schools. This “gold standard” 
evaluation design will allow the evaluation team to assess the impact of Diplomas Now on key 
predictors of students’ later graduation: attendance, behavior, and course performance. 

Furthermore, the evaluation is also studying the implementation of the Diplomas Now 
model. This report has presented findings about the first year of implementation in terms of the 
fidelity of model implementation and the nature of collaborations that undergird that implemen- 
tation. Complex, multifaceted whole-school reforms like Diplomas Now typically take a few 
years to reach full implementation. After the first year of implementation, analyses of imple- 
mentation fidelity — how similar the model is “on the ground” to the model as designed — 
suggest that overall, model implementation in the 32 DN schools has gotten moving and has 
gained traction. However, the DN schools have struggled to implement some of the aspects of 
the model that the Diplomas Now organizations believe are most critical. 

DN schools were most successful this first year in adapting a tiered intervention model 
and incorporating additional student support services like tutoring and after-school programs. 
This result may reflect the fact that both of these components offered new resources to the 
schools and came with extensive support from the Diplomas Now staff, circumstances likely to 
facilitate their implementation. Implementing these aspects of the model in a school required 
three distinct organizations to partner with each other — a notable accomplishment for the first 
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year of implementation, given that the local staff members of those organizations were partner- 
ing for the first time and needed to execute both shared, cross-organizational tasks and comple- 
mentary, organization-specific ones. However, DN schools saw less success introducing new 
curricula and peer coaching models. Both these components require that teachers and adminis- 
trators be willing to change a school’s culture and practices. Refonn of this nature is likely to 
require that the Diplomas Now staff invest time in building trust and understanding with a 
school’s staff before many of the school staffs members will welcome these new models. 
Before Diplomas Now, some of the schools had engaged in refomis with similar structural, 
curricular, or programmatic features, but survey data indicate that fewer than half had previous- 
ly adopted a refonn model created or managed by external organizations. This inexperience 
might explain some of the challenges they faced. 

As discussed in this chapter, confusion about the roles and responsibilities of Diplomas 
Now staff members and partner organizations may have hindered some aspects of implementa- 
tion during the first year. Clarifying and establishing roles and responsibilities as early as 
possible in the school year promotes cohesion and accountability among the Diplomas Now 
school-based staff members. Effectively communicating what Diplomas Now entails for a 
school and its personnel, and offering examples of successful outcomes — including site visits 
to other DN schools — may gamer the interest and commitment of administrators and teachers. 
Offering assistance to improve a school by doing “whatever needs to be done” is one way for 
Diplomas Now staff members to encourage administrators and teachers to engage with the 
model, but it can lead to staff members taking on tasks that have little to do with their real 
responsibilities. Depending on the school context, Diplomas Now staff members may need to 
seek an appropriate balance between building rapport and setting limits. 

The evaluation team will continue to study Diplomas Now implementation, and will 
provide an update about how implementation develops over time in the next report. In addition, 
the evaluation team will present findings about what changes Diplomas Now may be causing in 
DN schools compared with non-DN schools, and whether model implementation is beginning 
to have an impact on student and school staff attitudes and school-related behaviors. 
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Appendix A 

Baseline Administrator and Teacher Survey 
Response Rates and Measure Creation 




This appendix includes supplementary materials for the baseline administrator and teacher 
survey analyses found in Chapter 2. The first section presents the response rates for the baseline 
administrator and teacher surveys, and the second section describes the creation of measures 
used in the baseline survey analyses where more than one survey item was combined. 


Baseline Survey Response Rates 

The survey was administered online to teachers and administrators (principals and assistant 
principals) in all study schools during the fall of the first year of implementation (201 1 for wave 
1 schools and 2012 for wave 2 schools). The survey asked if respondents were working at the 
school during the prior school year. Those respondents who were present at the school during 
the prior year were then asked a variety of questions about their experiences at the school during 
that year (see the Data Collection Instrument Supplement for copies of the surveys). 1 Table A.l 
describes the overall response rate for each survey, among study schools assigned to implement 
the Diplomas Now model (DN schools) and study schools not assigned to implement it (non- 
DN schools). 

Some schools were omitted from these analyses due to missing data. Sixteen schools 
were dropped from the administrator survey analyses, including: 1) eight schools where none of 
the administrators who responded to the survey were present during the prior year, 2) five 
schools in random assignment blocks where all of the DN schools or all of the non-DN schools 
were dropped due to missing data, 2 and 3) three schools where data were lost after collection. 
Six schools were dropped from the teacher survey analyses, including: 1) two schools where 
none of the administrators who responded to the survey were present in the study school during 
the prior year, 2) one school in a random assigmnent block where all of the DN schools or all of 
the non-DN schools were dropped due to missing data, and 3) three schools where data were 
lost after collection. The second row in each panel of Table A.l lists the administrator and 
teacher response rates for the schools included in the analysis. Of these respondents, only 
administrators and teachers who were at the schools during the prior school year were included 
in the analyses. The final row in each panel lists the percentage of respondents who were 
working at the school during the prior school year. 


] The Data Collection Instrument Supplement to this report, Corrin et al. (2014), is available at 
www.mdrc.org. 

2 Since there was no comparison, the entire random assignment block was dropped. 
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Diplomas Now 
Table A.l 

Baseline Administrator and Teacher Survey Response Rates 


Rates (%) 

DN 

Schools 

Non-DN 

Schools 

All Study 
Schools 

Panel A: administrators 




Overall response rate 

83.3 

80.3 

81.9 

Sample size 

120 

117 

237 

Response rate among schools in the analysis sample 3 

85.7 

86.0 

85.9 

Survey respondents present during the prior school year* 5 

75.0 

73.8 

74.4 

Sample size 

98 

93 

191 

Panel B: teachers 




Overall response rates 

77.4 

79.5 

78.4 

Sample size 

1,677 

1,700 

3,377 

Response rates among schools in the analysis sample 3 

79.4 

83.6 

81.5 

Survey respondents present during the prior school year b t 

76.5 

78.1 

77.3 

Sample size 

1,625 

1,598 

3,223 


SOURCES: Baseline administrator and teacher surveys administered in fall 2011 (wave 1 schools) 
and fall 2012 (wave 2 schools). 

NOTES: a There were 16 schools dropped from the administrator survey analyses and 6 schools 
dropped from the teacher survey analyses due to missing data. 

b The baseline surveys were administered during the fall of the first implementation year, but the 
items used in the baseline analyses (Tables 2.4 through 2.7) asked administrators and teachers about 
their experiences during the prior school year. Only administrators and teachers that were at their 
schools during the prior year were included in these analyses. These values display the rate at which 
survey respondents were present in the prior school year among schools in the analysis sample. 

c There were 3 1 survey respondents who reported their roles at their schools as something other 
than teacher (that is, they were counselors or other school staff members). These respondents were 
excluded from this calculation and from the analysis. Seven of these respondents were from DN 
schools and 24 were from non-DN schools. 
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Baseline Survey Measure Creation and Factor Analysis 

Tables 2.4 through 2.7 in Chapter 2 compare DN and non-DN schools in measures of school 
structure, programming, curriculum, and culture in the year prior to the start of the evaluation. 
These tables present the average percentages of administrators and teachers in DN and non-DN 
schools who answered each item in the affirmative. In most cases, single survey items were 
used in the analyses. For a few measures, items were combined to create a construct. The 
following discussion describes those cases where responses across several items were averaged 
to create a combined measure. 

Administrator Survey Items 

Table 2.6 includes the administrator survey measure: “A school-wide behavior management 
system existed at the school.” The value for the construct was coded as 1 if the average of the 
following items was between 4 and 5 (“agree” to “strongly agree”); otherwise the value was 
coded as 0. (Eight items, Cronbach’s alpha = 0.88.) 

During the [prior] school year, to what extent would you agree or disagree with 
each of the following statements about your school? 

Teachers consistently enforced rules for student behavior in their classrooms. 

Teachers helped maintain discipline in the entire school, not just their own class- 
rooms (e.g., helping students transition between classes). 

The school had an effective system for providing positive reinforcement to stu- 
dents who met behavior expectations. 

The school had an effective system for responding to problem behaviors. 

The school had an effective system for tracking office referrals and problem be- 
haviors. 

Students received instruction and guidance on school discipline policies and pro- 
cedures. 

Educators conducted functional assessments and implemented individualized 
behavior plans as needed. 

Students with frequent disruptive behavior received a daily check in with an 
adult to monitor their progress in meeting behavior goals as needed. 

(Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly Agree) 

Table 2.7 includes the administrator survey measure: “Parents/guardians and communi- 
ty members were involved in school activities, helped make school decisions, and supported 
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school reform efforts.” The value for the construct was coded as 1 if the average of the follow- 
ing items was between 4 and 5 (“agree” to “strongly agree”); otherwise the value was coded as 
0. (Three items, Cronbach’s alpha = 0.91.) 

To what extent would you agree or disagree with each of the following state- 
ments about your school during the [prior] school year? 

Parents/guardians and community members were involved in school activities. 

Parents/guardians and community members were involved in decisions about 
school initiatives. 

Parents/guardians and community members supported school reform efforts 
(planned or implemented). 

(Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly Agree) 

Teacher Survey Items 

Table 2.6 includes the teacher survey measure: “Interdisciplinary teams reviewed data on 
individual students’ academic progress, course perfonnance, attendance, and behavior to 
identify students’ needs and determine appropriate responses.” The value for the construct was 
coded as 1 if the average of the following items was between 4 and 5 (“agree” to “strongly 
agree”); otherwise the value was coded as 0. (Five items, Cronbach’s alpha = 0.96.) 

To what extent do you agree or disagree with each of the following statements 
about your experiences at this school during the [prior] school year? 

Interdisciplinary teams reviewed data on individual student’s academic progress 
to identify student needs. 

Interdisciplinary teams reviewed data on individual student’s attendance to iden- 
tify student needs. 

Interdisciplinary teams reviewed data on individual student’s behavior to identify 
student needs. 

Interdisciplinary teams reviewed data on individual student’s course perfor- 
mance to identify student needs. 

Interdisciplinary teams determined approaches to respond to identified student 
needs. 

(Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly Agree) 
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Table 2.7 includes the teacher survey measure: “Teachers were involved in decisions 
about school policies, supported school refonn efforts, and in general, had positive morale.” 
The value for the construct was coded as 1 if the average of the following items was between 4 
and 5 (“agree” to “strongly agree”); otherwise the value was coded as 0. (Three items, 
Cronbach’s alpha = 0.82.) 

To what extent would you agree or disagree with each of the following state- 
ments about your experiences at this school during the [prior] school year? 

Teachers were involved in decisions about school policies. 

A majority of teachers supported school reform efforts (planned or implement- 
ed). 

In general, morale among staff was positive. 

(Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly Agree) 

Table 2.7 also includes the teacher survey item: “Students were respectful to peers and 
teachers, maintained academic integrity, took responsibility for their own learning, paid atten- 
tion, and did their schoolwork.” The value for the construct was coded as 1 if the average of the 
following items was between 4 and 5 (“agree” to “strongly agree”); otherwise the value was 
coded as 0. (Nine items, Cronbach’s alpha = 0.92.) 

To what extent would you agree or disagree with each of the following state- 
ments about experiences with your students during the [prior] school year? 

Students communicated with their peers in a respectful manner most of the time. 

Students communicated with me in a respectful manner most of the time. 

Students took responsibility for their own learning most of the time. 

Students maintained academic honesty on tests or written exams most of the 
time. 

Students appeared to take pride in their schoolwork most of the time. 

Students put forth effort to understand difficult material most of the time. 

Students appeared to pay attention during my instruction most of the time. 

Students remained on task during self-directed activities most of the time. 

Students completed their assignments most of the time. 

(Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly Agree) 
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Appendix B 

Program and School Staff Survey 
Response Rates and Fidelity Matrix 




This appendix includes supplementary information on the program and school staff surveys and 
on the fidelity matrix, both discussed in Chapter 3. Table B.l provides the response counts for 
each of the program staff surveys (Diplomas Now Implementation Support Team, school 
transfonnation facilitator, City Year program manager, and Communities In Schools site 
coordinator) and the counts of respondents to the follow-up administrator and teacher surveys. 
Table B.2 is the fidelity matrix used for the analyses found in Tables 3.3 through 3.5. 


113 



114 


Diplomas Now 
Table B.l 


First-Year Follow-Up Program and School Staff Survey Response Counts 



DN Middle Schools 

DN High Schools 

All DN Schools 

Data Sources 

Total 

Respondents 

Total 

Schools 

Total 

Respondents 

Total 

Schools 

Total 

Respondents 

Total 

Schools 

Program staff surveys 

Diplomas Now Implementation Support Team survey 

17 

17 

15 

15 

32 

32 

School transformation facilitator survey 

17 

17 

15 

15 

32 

32 

City Year program manager survey 

17 

17 

15 

15 

32 

32 

Communities In Schools site coordinator 

17 

17 

15 

15 

32 

32 

survey and service records 
School staff surveys 3 

Administrator surveys 

38 

16 

56 

15 

94 

31 

All 6th- and 9th-grade teacher surveys 

254 

17 

488 

15 

742 

32 

Core 6th- and 9th-grade teacher surveys 

172 

17 

313 

15 

485 

32 

6th- and 9th-grade English/language arts teacher surveys 

76 

17 

95 

15 

171 

32 

6th- and 9th-grade math teacher surveys 

57 

16 

90 

14 

147 

30 

Sample size 

NA 

17 

NA 

15 

NA 

32 


SOURCES: Diplomas Now fidelity of implementation program staff surveys administered in spring 2012 and 2013, and school staff (administrator 
and teacher) surveys administered in spring 2012 and 2013. 

NOTES: "Core" academic areas are identified as math. English/language arts, sciences, and social studies. 

Two middle schools consisted only of seventh and eighth grade and did not include a sixth grade. In these instances seventh-grade teacher data 
were used in lieu of sixth-grade data. 







Diplomas Now 
Table B.2 

Diplomas Now Fidelity of Implementation Matrix 


Component 

Operational Definition Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar I. Teacher Teams and Small Learning Communities: Strong Learning Environments 




Site-based team standards 

Site-based team (administrator, school STF 
transformation facilitator, project 
manager, and site coordinator) 
standards for collaboration, 
communication, and decision making 

0: Not in place 
1: Partially/in process 
2: In place 

1 = Adequate 

2 = High fidelity 

No 

96.9 

Site-based team meetings 

Brief meetings for site-based team to STF 
review program implementation 
(approx. 30 minutes) 

0: Once a month or less 
1 : Biweekly 
2: Weekly or more 
frequently 

1 = Adequate 

2 = High fidelity 

Yes 

78.1 

4x4 block 
(high school only) 

4 class periods of 75 to 90 minutes that STF 
meet daily (or at least 4 days a week) 

0: No 

1 : Hybrid/acceptable 
alternative 
2: Yes 

1 = Adequate 

2 = High fidelity 

Yes 

86.7 

Extended class periods 
(middle school only) 

70- to 90-minute class periods for core STF 
academic classes 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

Yes 

52.9 

Small learning 
communities 

Interdisciplinary teams of teachers who STF 
work with the same small group of 
students 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

Yes 

68.8 
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Table B.2 (continued) 


Component 

Operational Definition 

Source 3 Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar I. Teacher Teams and Small Learning Communities: Strong Learning Environments 




Interdisciplinary teacher 
team meetings 

Meetings where interdisciplinary core 
teachers discuss shared students 

STF 0: Do not/rarely occur 

1 : Occur monthly 
2: Occur biweekly 
3: Occur weekly 
4: Occur multiple times a 
week 

5 : Occur daily 

4 = Adequate 

5 = High fidelity 

Yes 

25.0 


Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 


9th-grade success 
academy 

(high school only) 

Separate academy for 9th-graders with STF 
its own administrators, teachers, 
counselors, etc. 

0: Not offered 
1 : Offered 

1 = Adequate/ 
high fidelity 

Yes 

60.0 

School climate program 
(middle school only) 

Reform program on school climate for STF 
middle school students 

0: Not offered 

1 : Offered in 6th grade 

only 

2: Offered in both 6th and 
7 th grades 

3: Offered in 6th, 7th, and 
8th erades 

1 = Adequate 

2 = High fidelity 

No 

41.2 

Mastering the Middle 
Grades 

(middle school only) 

Reform curriculum on school success STF 
skills for middle school students 

0: Not offered 

1 : Offered in 6th grade 

only 

2: Offered in both 6th and 
7 th grades 

3: Offered in 6th, 7th, and 
8th grades 

1 = Adequate 

2 = High fidelity 

No 

11.8 


(continued) 



Table B.2 (continued) 


Percentage of 
Critical Schools 


Component Operational Definition Source 3 Fidelity Scale Criterion Component Adequate 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 


Freshman seminar 
(high school only) 

Seminar offered to 9th-grade students 
during their first semester 

STF 

0: Not offered 
1: Offered 

1 = Adequate/ 
high fidelity 

Yes 

73.3 

Freshman seminar 
availability 
(high school only) 

Number of units of the freshman 
seminar offered to 9th-grade students 

STF 

0: none 
1: 1 unit 
2: 2 units 
3: 3 units 
8: 8 units 

6 = Adequate 
8 = High fidelity 

No 

46.7 

Common Core State 
Standards in mathematics 
(middle school only) 

High implementation of Common Core STF 
Standards in mathematics 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

76.5 

Evidence-based 
mathematics curriculum 
(middle school only) 

Reform mathematics curriculum for 
middle school students 

STF 

0: Not offered 
1 : Not offered, but an 
adequate alternative 
offered 
2: Offered 

1 = Adequate 

2 = High fidelity 

No 

58.8 

Transitions to Advanced 

Mathematics 

(high school only) 

Reform mathematics curriculum for 
9th-grade students 

STF 

0: Not offered 
1: Offered 

1 = Adequate/ 
high fidelity 

No 

60.0 

Transitions to Advanced 
Mathematics availability 
(high school only) 

Classes that meet at least 4 times per 
week for at least 70 minutes per 
meeting 

STF 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

46.7 


(continued) 


Table B.2 (continued) 


Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 

Geometry Foundations 
(high school only) 

Reform mathematics curriculum for 
lOth-grade students 

STF 

0: Not offered 
1 : Offered 

1 = Adequate/ 
high fidelity 

No 

13.3 

Geometiy Foundations 

availability 

(high school only) 

Classes that meet at least 4 times per 
week for at least 70 minutes per 
meeting 

STF 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

13.3 

Algebra Foundations II 
(high school only) 

Reform mathematics curriculum for 
1 lth-grade students 

STF 

0: Not offered 
1 : Offered 

1 = Adequate/ 
high fidelity 

No 

0.0 

Algebra Foundations II 

availability 

(high school only) 

Classes that meet at least 4 times per 
week for at least 70 minutes per 
meeting 

STF 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

0.0 

Computer and Team 
Assisted Mathematics 
Acceleration (CATAMA) 
(middle school only) 

Reform mathematics curriculum that 
provides additional instruction and 
support to underprepared middle 
school students 

STF 

0: Not offered 
1 : Not offered, but an 
adequate alternative 
offered 
2: Offered 

1 = Adequate 

2 = High fidelity 

No 

29.4 

CATAMA 
(high school only) 

Reform mathematics curriculum that 
provides additional instruction and 
support to underprepared high school 
students 

STF 

0: Not offered 
1 : Not offered, but an 
adequate alternative 
offered 
2: Offered 

1 = Adequate 

2 = High fidelity 

No 

20.0 
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Percentage of 






Critical 

Schools 

Component 

Operational Definition 

Source 2 

Fidelity Scale 

Criterion 

Component 

Adequate 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 

Student Team Literature 

Reform English/language arts 

STF 

0: Not offered 

1 = Adequate 

Yes 

41.2 

(middle school only) 

curriculum for middle school students 


1 : Not offered, but an 
adequate alternative 
offered 
2: Offered 

2 = High fidelity 



Savvy Readers' Lab 

Reform English/language arts 

STF 

0: Not offered 

1 = Adequate 

Yes 

41.2 

(middle school only) 

curriculum for middle school students 


1 : Not offered, but an 
adequate alternative 
offered 
2: Offered 

2 = High fidelity 



Strategic Reading 

Reform reading curriculum for 9th- 

STF 

0: Not offered 

1 = Adequate/ 

No 

60.0 

(high school only) 

grade students 


1 : Offered 

high fidelity 



Strategic Reading 

Classes that meet at least 4 times per 

STF 

0: No 

1 = Adequate/ 

No 

46.7 

availability 

week for at least 70 minutes per 


1: Yes 

high fidelity 



(high school only) 

meeting 






Reading and Writing in 

Reform reading and writing curriculum STF 

0: Not offered 

1 = Adequate/ 

No 

0.0 

Your Career 

for 1 Oth-grade students 


1 : Offered 

high fidelity 



(high school only) 
Reading and Writing in 

Classes that meet at least 4 times per 

STF 

0: No 

1 = Adequate/ 

No 

0.0 

Your Career availability 

week for at least 70 minutes per 


1: Yes 

high fidelity 



(high school only) 

meeting 






College Prep Reading and Reform reading and writing curriculum STF 

0: Not offered 

1 = Adequate/ 

No 

0.0 

Writing 

(high school only) 

for 1 lth-grade students 


1 : Offered 

high fidelity 




(continued) 



Table B.2 (continued) 


Component Operational Definition Source 3 Fidelity Scale Criterion 

Pillar II. Curriculum and Instruction with Professional Development: Curriculum for College Readiness 


Critical 

Component 


College Prep Reading and Classes that meet at least 4 times per STF 
Writing availability (high week for at least 70 minutes per 
school only) meeting 

Accelerating Literacy for Intensive literacy support for 9th-grade STF 
Adolescents (ALFA) Lab students who are significantly below 
(high school only) grade level 


Pillar II. Curriculum and Instruction with Professional Development: 

Freshman seminar teacher Professional development and job- STF 
support (high school only) embedded support for freshman 
seminar teachers 


0: No 1 = Adequate/ 

l:Yes high fidelity 

0: Not offered 1 = Adequate 

1 : Not offered, but an 2 = High fidelity 

adequate alternative 
offered 
2: Offered 

Professional Development and Peer Coaching 


Instructional coaching for Periods of support per week provided STF 
mathematics teachers to mathematics teachers by math 
coaches 


Instructional coaching 
cycle completed with 
mathematics teachers 


Instructional coaching cycle (plan, 
coteach, model, and debrief) 
completed by mathematics coach with 
teachers 


0: Not provided 
1 : Provided 

2: Provided by school and 
student support services 
(S4) facilitator 
3: Provided by school 
transformation facilitator 
0: Less than 1 period per 
teacher 

1 : 1 period per teacher 
2: 2 periods per teacher 
0: Did not occur 
1 : Occurred annually 
2: Occurred semiannually 
3 : Occurred monthly 
4: Occurred biweekly 
5: Occurred weekly 


1 -2 = Adequate 
3 = High fidelity 


1 = Adequate 

2 = High fidelity 


4 = Adequate 

5 = High fidelity 


(continued) 


Table B.2 (continued) 






Critical 

Percentage of 
Schools 

Component 

Operational Definition Source 3 

Fidelity Scale 

Criterion 

Component 

Adequate 

Pillar II. Curriculum and Instruction with Professional Development: Professional Development and Peer Coaching 

Instructional coaching for 

Periods of support per week provided STF 

0: Less than 1 period per 

1 = Adequate 

Yes 

62.5 

English/language arts 

to English/language arts teachers by 

teacher 

2 = High fidelity 



teachers 

English/language arts coaches 

1 : 1 period per teacher 
2: 2 periods per teacher 




Instructional coaching 

Instructional coaching cycle (plan, STF 

0: Did not occur 

4 = Adequate 

No 

40.6 

cycle completed with 

coteach, model, and debrief) 

1 : Occurred annually 

5+ = High 



English/language arts 

completed by English/language arts 

2: Occurred semiannually 

fidelity 



teachers 

coach with teachers 

3 : Occurred monthly 
4: Occurred biweekly 
5 : Occurred weekly 




Pillar III. Tiered Student Supports: Tiered Intervention Model 

Coordinated Early 

Data system that tracks student STF 

0: No EWI system in place 

1 = Adequate 

Yes 

93.8 

Warning Indicator (EWI) 

attendance, behavior, and course 

1 : In place but needs 

2 = High fidelity 



system 

performance and alerts teachers as 

improvement 





students begin to fall off the graduation 

2: In place and is timely, 





track 

complete, accurate, and 
available 




Collaborative 

Interdisciplinary teacher meetings to STF 

0: Occurred monthly or 

1 = Adequate 

Yes 

90.6 

interdisciplinary EWI 

discuss students demonstrating off- 

less 

2 = High fidelity 



team meetings 

track indicators 

1 : Occurred biweekly 
2: Occurred weekly 




Tiered intervention model 

A plan for integrating the use of the STF 

0: Not in place 

1 = Adequate/ 

No 

75.0 


EWI system and 
scheduling EWI meetings 

1 : In place 

high fidelity 
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Percentage of 






Critical 

Schools 

Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Component 

Adequate 

Pillar III. Tiered Student Supports: Student Supports 

Mathematics classroom 

Additional in-class support from City 

PM 

0: No 

1 = Adequate/ 

Yes 

100.0 

support 

Y ear corps members offered in 
mathematics classrooms year-round 


1: Yes 

high fidelity 



Mathematics classroom 

Percentage of City Year corps 

PM 

0: 0-24% 

3 = Adequate 

Yes 

62.5 

support: percentage of 

members embedded in mathematics 


1: 25-49% 

4 = High fidelity 



City Year corps 

classrooms 


2: 50-74% 







3: 75-99% 







4: 100% 




Mathematics classroom 

Frequency of City Year corps 

PM 

0: Does not occur 

3 = Adequate 

Yes 

100.0 

support frequency 

members in mathematics classrooms: 


1 : 1 time per week 

4 = High fidelity 




weekly average 


2: 2 times per week 
3: 3 times per week 







4: 4 times per week 




Mathematics classroom 

Percentage of math classrooms with 

PM 

0: 0-24% 

3 = Adequate 

Yes 

40.6 

support: percentage of 

embedded City Year corps members 


1: 25-49% 

4 = High fidelity 



classrooms 



2: 50-74% 







3: 75-99% 







4: 100% 




English/language arts 

Additional in-class support from City 

PM 

0: No 

1 = Adequate/ 

Yes 

100.0 

classrooms support 

Y ear corps members offered in 
English/language arts classrooms year- 
round 


1: Yes 

high fidelity 



English/language arts 

Percentage of City Year corps 

PM 

0: 0-24% 

3 = Adequate 

Yes 

65.6 

classrooms support: 

members embedded in 


1: 25-49% 

4 = High fidelity 



percentage of City Year 

English/language arts classrooms 


2: 50-74% 




corps 



3: 75-99% 







4: 100% 
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Table B.2 (continued) 


Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar III. Tiered Student Supports: Student Supports 






English/language arts 
classrooms support 
frequency 

Frequency of City Year corps 
members in English/language arts 
classrooms: weekly average 

PM 

0: Does not occur 
1 : 1 time per week 
2: 2 times per week 
3: 3 times per week 
4: 4 times per week 

3 = Adequate 

4 = High fidelity 

Yes 

100.0 

English/language arts 
classroom support: 
percentage of classrooms 

Percentage of English/language arts 
classrooms with embedded City Year 
corps members 

PM 

0: 0-24% 
1: 25-49% 
2: 50-74% 
3: 75-99% 
4: 100% 

3 = Adequate 

4 = High fidelity 

Yes 

50.0 

Mathematics tutoring 
stmcture 

Mathematics tutoring stmcture and 
schedule 

PM 

0: Not in place 
1: In place 

1 = Adequate/ 
high fidelity 

Yes 

71.9 

City Year corps 
integration into 
mathematics tutoring 
structure 

Integration of City Year corps 
members into mathematics tutoring 
stmcture 

PM 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

Yes 

62.5 

Mathematics tutoring: 
number of students 

Number of students receiving 
mathematics tutoring per City Y ear 
corps member 

PM 

0: 0 students 
1:1-3 students 
2: 4-6 students 
3: 7-10 students 
4: More than 10 students 

3 = Adequate 

4 = High fidelity 

Yes 

50.0 

Literacy tutoring structure 

Literacy tutoring stmcture and 
schedule 

PM 

0: Not in place 
1 : In place 

1 = Adequate/ 
high fidelity 

Yes 

75.0 

City Year corps 
integration into literacy 
tutoring stmcture 

Integration of City Year corps 
members into literacy tutoring 
stmcture 

PM 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

Yes 

59.4 
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Table B.2 (continued) 


Component 

Operational Definition Source 2 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar III. Tiered Student Supports: Student Supports 





Literacy tutoring: number 
of students 

Number of students receiving literacy PM 
tutoring per City Year corps member 

0: 0 students 
1:1-3 students 
2: 4-6 students 
3: 7-10 students 
4: More than 10 students 

3 = Adequate 

4 = High fidelity 

Yes 

59.4 

Attendance support 
structure 

Integration of City Year corps PM 

members into school attendance 
program 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

Yes 

71.9 

Attendance coaching 

Number of students receiving PM 

attendance coaching per City Year 
corps member 

0: 0 students 
1:1-3 students 
2: 4-6 students 
3: 7-10 students 
4: More than 10 students 

1 = Adequate 

2 = High fidelity 

Yes 

93.8 

Behavior coaching 

Number of students receiving behavior PM 
coaching per City Y ear corps member 

0: 0 students 
1:1-3 students 
2: 4-6 students 
3: 7-10 students 
4: More than 10 students 

1 = Adequate 

2 = High fidelity 

Yes 

93.8 

After-school program 

After-school program or extended PM 

learning time for subset of school’s 
students 

0: Not offered 
1 : Offered 

1 = Adequate/ 
high fidelity 

Yes 

96.9 

After-school program 
recruitment 

Consistent recruitment by City Y ear PM 
corps members of students not on track 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

78.1 


to graduate to attend after-school 
program 
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Table B.2 (continued) 


Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar III. Tiered Student Supports: Student Supports 






After-school program 
attendance rate 

Percentage of students who regularly 
attend after-school program 

PM 

0: 0% 

1 : Less than 25% 

2: Greater than 25% 

2 = Adequate/ 
high fidelity 

No 

56.3 

City Year after-school 
program duration 

Duration of City Year after-school 
program 

PM 

0: No program 
1 : present 14 of year 
2: Present 'A of year 
3 : Present % of year 
4: Present all year 

4 = Adequate/ 
high fidelity 

No 

46.9 

Existing after-school 
program integration 

Integration of City Year corps 
members into existing after-school 
program 

PM 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

40.6 

Existing after-school 
program duration 

Duration of existing after-school 
program where City Year corps 
members are integrated 

PM 

0: No program 
1 : present 14 of year 
2: Present 'A of year 
3 : Present % of year 
4: Present all year 

4 = Adequate/ 
high fidelity 

No 

43.8 

Whole-school activities 

Number of whole-school activities (for 
example, health fairs or career days) 
during the year 

PM 

0: 0 activities 
1: 1 activity 
2: 2 activities 
3: 3 activities 
4: 4 activities 
5: 5 activities 
6: 6 activities 
7: 7 activities 
8: 8+ activities 

4 = Adequate 
8 = high fidelity 

Yes 

100.0 
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Percentage of 






Critical 

Schools 

Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Component 

Adequate 

Pillar III. Tiered Student Supports: Student Case Management 

Case management of Tier 

Percentage of EWI system Tier III 

CIS 

0: 0-24% 

If less than 100 

Yes 

71.9 

III students 

students case managed 


1: 25-49% 

Tier III students: 






2: 50-74% 

4 = Adequate/ 






3: 75-99% 

high fidelity 






4: 100% 

If greater than 
100 Tier III 
students: 

3 = Adequate 

4 = High fidelity 



Case management of Tier 

Percentage of case-managed students 

CIS 

0: 0-24% 

3 = Adequate 

Yes 

90.6 

III students: individual 

with individual student plans 


1: 25-49% 

4 = High fidelity 



student plan 



2: 50-74% 







3: 75-99% 







4: 100% 




Case management of Tier 

Percentage of case-managed students 

CIS 

0: 0-24% 

3 = Adequate 

Yes 

75.0 

III students: attendance 

identified with attendance issues who 


1: 25-49% 

4 = High fidelity 




are provided with attendance 


2: 50-74% 





interventions 


3: 75-99% 







4: 100% 




Case management of Tier 

Percentage of case-managed students 

CIS 

0: 0-24% 

3 = Adequate 

Yes 

90.6 

III students: behavior 

identified with behavior problems who 


1: 25-49% 

4 = High fidelity 




are provided with behavior 


2: 50-74% 





interventions 


3: 75-99% 







4: 100% 
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Percentage of 






Critical 

Schools 

Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Component 

Adequate 

Pillar III. Tiered Student Supports: Student Case Management 

Case management of Tier Percentage of case-managed students 

CIS 

0: 0-24% 

3 = Adequate 

Yes 

65.6 

III Students: academic 

identified with academic needs who 


1: 25-49% 

4 = High fidelity 



needs 

are provided with academic assistance 


2: 50-74% 





interventions 


3: 75-99% 







4: 100% 




Case management of Tier Percentage of case-managed students 

CIS 

0: 0-24% 

3 = Adequate 

No 

87.5 

III students: basic needs 

identified with basic needs (for 


1: 25-49% 

4 = High fidelity 




example, for food, clothing, or shelter) 


2: 50-74% 





who are provided with basic 


3: 75-99% 





needs/resource interventions 


4: 100% 




Percentage of students 

Percentage of overall student 

CIS 

If school population is 

2 = Adequate 

No 

65.6 

case managed 

population case managed 


1 ,000 or greater: 

3+ = High 






0: None 
1: 1-4% 

2: 5-10% 

3: 11-15% 

If school population is less 
than 1,000: 

0: None 
1: 1-9% 

fidelity 






2: 10-15% 







3: 16-20% 




Whole-school services: 

Enrichment/motivation services 

CIS 

0: Not provided 

1 = Adequate/ 

No 

96.9 

enrichment/motivation 



1 : Provided 

high fidelity 



Whole-school services: 

F amily engagement/strengthening 

CIS 

0: Not provided 

1 = Adequate/ 

No 

84.4 

family engagement 

services 


1 : Provided 

high fidelity 
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Percentage of 







Critical 

Schools 

Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Component 

Adequate 

Pillar III. Tiered Student Supports: Student Case Management 

Whole-school services: 

Life skills/social development services 

CIS 

0: Not provided 

1 = Adequate/ 

No 

75.0 

life skills 



1 : Provided 

high fidelity 



Whole-school services: 

College/career services 

CIS 

0: Not provided 

1 = Adequate/ 

No 

68.8 

college/career 



1 : Provided 

high fidelity 



Whole-school services: 

Professional physical health services 

CIS 

0: Not provided 

1 = Adequate/ 

No 

46.9 

physical health 



1 : Provided 

high fidelity 



Whole-school services: 

Community service opportunities 

CIS 

0: Not provided 

1 = Adequate/ 

No 

37.5 

community service 



1 : Provided 

high fidelity 



Whole-school services: 

Professional mental health services 

CIS 

0: Not provided 

1 = Adequate/ 

No 

37.5 

mental health 



1 : Provided 

high fidelity 



Pillar IV. Can-Do Culture and Climate: Integrated On-Site Support 

Mathematics instructional 

Technical assistance from the 

STF 

0: Not provided to school 

1 = Adequate/ 

Yes 

100.0 

facilitator 

mathematics instructional facilitator 


1 : Provided 

high fidelity 




English/language arts 
instructional facilitator 

Technical assistance from the 
English/language arts instructional 
facilitator 

STF 0: Not provided to school 

1 : Provided 

1= Adequate/ Yes 

high fidelity 

100.0 

School and student 
support services (S4) 

Technical assistance from S4 
facilitator 

STF 0: Not provided to school 

1 : Provided 

1 = Adequate/ Yes 

high fidelity 

100.0 


facilitator 


(continued) 
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Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar IV. Can-Do Culture and Climate: Inteerated On-Site SuDDort 





School transformation 
facilitator start date 

Date school transformation facilitator 
began serving the school 

STF 

0: After the first progress 
report 

1 : After the first day of 
school, but before the first 
progress report 
2 : After Summer Institute, 
but before the first day of 
school 

3: Before Summer Institute 

2 = Adequate 

3 = High fidelity 

No 

81.2 

Mathematics instructional 
coach 

Amount of time mathematics coach 
worked at school 

STF 

0: No mathematics coach 
in place 

1 : Less than half time at 
school 

2: At least half time at 
school 

3 : Full time at school 

2 = Adequate 

3 = High fidelity 

Yes 

75.0 

English/language arts 
instructional coach 

Amount of time English/language arts 
coach worked at school 

STF 

0: No English/language 
arts coach in place 
1 : Less than half time at 
school 

2: At least half time at 
school 

3 : Full time at school 

2 = Adequate 

3 = High fidelity 

Yes 

81.3 

Site coordinator 

Site coordinator at the school 

CIS 

0: Not in place 
1 : In place 

1 = Adequate/ 
high fidelity 

Yes 

100.0 
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Component 


Operational Definition 


Source 8 Fidelity Scale 


Criterion 


Percentage of 
Critical Schools 

Component Adequate 


Pillar IV. Can-Do Culture and Climate: Integrated On-Site Support 


Site coordinator start date Date site coordinator began serving 
school 


City Year corps members 
retention 


Percentage of City Year corps 
members retained 


CIS 0: After the first progress 1= Adequate Yes 

report 2 = High fidelity 

1 : After the first day of 
school, but before the first 
progress report 
2: After Summer Institute, 
but before the first day of 
school 

3: Before Summer Institute 

CY 0:0-24% 3= Adequate Yes 

1 : 25-49% 4 = High fidelity 

2: 50-74% 

3: 75%-89% 

4: 90% or more 


81.3 


100.0 


City Year corps members: Ratio of City Year corps members to PM 
ratio to students students 


City Year corps members: Ratio of second -year City Year corps PM 
ratio of second -year to members to first-year City Year corps 
first-year members 


0: No City Year corps 
members 
1: 1:50 or higher 
2: 1:30-49 
3: 1:20-29 

0: Greater than 1:10 
1: 1:10 

2: Less than 1:10 


2= Adequate Yes 
3 = High fidelity 


1 = Adequate No 

2 = High fidelity 


78.1 


78.1 
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Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Pillar IV. Can-Do Culture and Climate: Family and Community Involvement 




Workshops for parents 

Frequency of workshops offered to 
parents 

STF 

0: Never 
1 : Once 

2: 2-3 times per year 
3: Quarterly 

1 = Adequate 

2 = High fidelity 

No 

87.5 

Information for parents 

Frequency with which teachers sent 
information to parents on how to help 
their children 

STF 

0: Never 
1 : Once 

2: 2-3 times per year 
3: Quarterly 

1 = Adequate 

2 = High fidelity 

No 

78.1 

Parent-teacher 

conferences 

Parent-teacher conferences (with all 
core teachers) scheduled for each 
student’s family 

STF 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

75.0 

Action team for 
partnerships 

Action team for partnerships, or its 
equivalent, to develop, implement, and 
evaluate work on family and 
community engagement 

STF 

0: No 

1: Yes, but the team did 
not receive training 
2: Yes and the team did 
receive training 

1 = Adequate 

2 = High fidelity 

No 

56.3 

Parent volunteers 

Parents formally recruited and trained 
to work as school volunteers 

STF 

0: No 
1: Yes 

1 = Adequate/ 
high fidelity 

No 

18.8 

Parent engagement 
practices 

How well actual parent engagement 
practices matched ideal practices 

STF 

0: Weakly 
1: Somewhat 
2: Well 
3: Very well 

1 = Adequate 

2 = High fidelity 

Yes 

65.6 


(continued) 



Table B.2 (continued) 


Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Program Staff Training and Development 






Partner responsiveness: 
Community In Schools to 
Talent Development 

Talent Development staffs access to 
Communities In Schools' staff on a 
timely basis as needed 

STF 

0: None 
1 : A little 
2: Some 
3 : Quite a bit 

1 = Adequate 
3 = High fidelity 

Yes 

100.0 

Partner responsiveness: 
City Year to Talent 
Development 

Talent Development staffs access to 
City Year's staff on a timely basis as 
needed 

STF 

0: None 
1 : A little 
2: Some 
3 : Quite a bit 

1 = Adequate 
3 = High fidelity 

Yes 

100.0 

Partner responsiveness: 
Talent Development to 
Communities In Schools 

Communities In Schools staffs access 
to Talent Development's staff on a 
timely basis as needed 

SC 

0: None 
1 : A little 
2: Some 
3 : Quite a bit 

1 = Adequate 
3 = High fidelity 

Yes 

100.0 

Partner responsiveness: 
City Year to Communities 
In Schools 

Communities In Schools staffs access 
to City Year's staff on a timely basis as 
needed 

SC 

0: None 
1 : A little 
2: Some 
3 : Quite a bit 

1 = Adequate 
3 = High fidelity 

Yes 

100.0 

Partner responsiveness: 
Communities In Schools 
to City Year 

City Year corps members' access to 
Communities In Schools' staff on a 
timely basis as needed 

PM 

0: None 
1 : A little 
2: Some 
3 : Quite a bit 

1 = Adequate 

2 = High fidelity 

Yes 

93.8 

Partner responsiveness: 
Talent Development to 
City Year 

City Year corps members' access to 
Talent Development's staff on a timely 
basis as needed 

PM 

0: None 
1 : A little 
2: Some 
3 : Quite a bit 

1 = Adequate 

2 = High fidelity 

Yes 

93.8 
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Table B.2 (continued) 


Component Operational Definition 

Source 2 Fidelity Scale 

Criterion 

Percentage of 
Critical Schools 

Component Adequate 

Program Staff Training and Development 


Talent Development in- 
service training 


Diplomas Now in-service 
training 


Diplomas Now Summer 
Institute 


Professional development plan for on- DNIST 
site training for teachers and school- 
based staff members focused on 
curriculum, school climate, and 
teaming 

Professional development plan for DNIST 
school-based team to engage in 
continuing professional development 
opportunities throughout the school 
year 

Three-day summer training session for DNIST 
school-based team aimed at helping a 
new Diplomas Now school start strong 


0: Missing or not followed 1 = Adequate 
1 : Partially implemented 2 = High fidelity 
2: Fully implemented 


0: Missing or not followed 1 = Adequate 
1 : Partially implemented 2 = High fidelity 
2: Fully implemented 


0: Not attended and no 
alternate training provided 
1 : Not attended, but partial 
alternate training provided 
2: Attended or received Hill- 
fledged alternate training 


Yes 


81.3 


Yes 


78.1 


1 = Adequate 

2 = High fidelity 


Yes 


78.1 


Kickoff planning sessions Joint planning sessions for school 

administrators and teachers prior to the 
start of the school year 


DNIST 0: No 
1: Yes 

2: Yes, in April/May 


1 = Adequate 

2 = High fidelity 


Yes 


59.4 


City Year corps member 
training: teacher 


City Year corps members' participation PM 
in relevant teacher professional 


professional development development opportunities 


0: Never 

1 : Sometimes (once or 
twice) 

2: Consistently 


1= Adequate Yes 
2 = High fidelity 


93.8 
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Table B.2 (continued) 


Component 

Operational Definition 

Source 3 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Program Staff Training and Development 






School transformation 
facilitator summer 
training session 

Five-day summer training session for 
school transformation facilitators 

DNIST 

0: Not attended and no 
alternate training provided 
1 : Not attended, but 
alternate training provided 
2: Attended 

1 = Adequate 

2 = High fidelity 

Yes 

96.9 

Mathematics Coaches 
Institute 

Three-day summer training session for 
math coaches 

STF 

0: Not attended and no 
alternate training provided 
1 : Not attended, but 
alternate training provided 
2: Attended 

1 = Adequate 

2 = High fidelity 

No 

43.8 

English/language arts 
Coaches Institute 

Three-day summer training session for 
English/language arts coaches 

STF 

0: Not attended and no 
alternate training provided 
1 : Not attended, but 
alternate training provided 
2: Attended 

1 = Adequate 

2 = High fidelity 

No 

46.9 

Site coordinator 
certification program 

1 1 -module online course, 
approximately 1.5-2 hours per module, 
for site coordinators 

CIS 

0: Did not complete 
1 : Completed 4 courses 
2: Completed 8 courses 
3: Completed all courses 
4: Completed certification 
prior to the start of the 
school year 

3 = Adequate 

4 = High fidelity 

Yes 

40.6 

City Year corps members 
training: use of data 

Training for City Year corps members 
in the use of data to identify 

PM 

0: Not received 
1 : Received 

1 = Adequate/ 
high fidelity 

Yes 

56.3 


interventions 
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Table B.2 (continued) 


Component 

Operational Definition 

Source 2 

Fidelity Scale 

Criterion 

Critical 

Component 

Percentage of 
Schools 
Adequate 

Program Staff Training and Development 

City Year corps members 
continuing coaching and 
support 

Continuing support for City Year corps PM 
members in the use of data to identify 
interventions 

0: Not received 
1 : Received 

1 = Adequate/ 
high fidelity 

Yes 

53.1 

City Year corps member 
training: EWI 

EWI-related training sessions for City 
Year corps members throughout the 
year 

PM 

0: Never 
1 : Once 

2: Once a month 
3: Twice a month 
4: Three times a month 
5 : Four times a month or 
more 

4 = Adequate 

5 = High fidelity 

No 

34.4 


NOTES: Average percentages of sites adequate for middle and high school-specific components reflect only the appropriate number of middle (17) 
or high (15) schools. 

“Source surveys were abbreviated as follows: DNIST = the Diplomas Now Implementation Support Team survey; STF = Talent Development 
school transformation facilitator survey; PM = City Year program manager survey; SC = Communities In Schools site coordinator survey; CY = 
City Year records; CIS = Communities In Schools records. 




Appendix C 

Interview and Focus-Group Response Rates 
and Case-Study Methodology 




This appendix includes supplementary information on the qualitative data (from interviews and 
focus groups) collected from case-study sites and discussed in Chapters 3 and 4. The first 
section consists of Table C.l, which shows the numbers of participants in the interviews and 
focus groups. The second section describes the methodology of the qualitative research. The 
final section consists of Table C.2, which adds more detail to Table 3.6, showing the frequen- 
cies of various types of responses by respondent type (Diplomas Now staff, school/district staff, 
or parent/student) and school type (middle or high school). 


Interview and Focus-Group Response Counts 


Diplomas Now 
Table C.l 


Numbers of Interview and Focus-Group Participants from Case-Study Sites 



All DN Schools 

DN Middle Schools 

DN High Schools 

Total Total 
Participants Schools 

Total 

Participants 

Total 

Schools 

Total Total 
Participants Schools 

Respondent type 






Diplomas Now staff members 

68 

7 

36 

3 

32 4 

District/school staff members 

50 

6 

24 

3 

25 3 

Parents 

14 

4 

7 

2 

7 2 

Students 

41 

4 

21 

2 

20 2 

Sample size 

173 

7 

88 

3 

84 4 


SOURCES: Diplomas Now program staff, school, and district interviews and focus groups, spring 2012 
and 2013. 

NOTES: Data were collected at seven DN schools representing four districts (three schools and 15 
participants in spring 2012 and four schools and 158 participants in spring 2013). The total number of 
district/school staff participants is higher than the sum of the middle and high school participants because 
one interview was conducted with a district-level staff member who oversaw both DN middle schools and 
DN high schools, and who was not included in either separate count. 


Case-Study Methodology 

To ensure the consistent use of best practices, the research team used standardized processes 
and detailed audit trails to collect and analyze case-study data. The following sections provide 
detail regarding protocol development, on-site data-collection activities, and qualitative analysis 
procedures. 
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Protocol Development 

The research team paid careful attention to the development of protocols for semistruc- 
tured interviews and focus groups intended to yield in-depth information about participants’ 
experiences during the first year of implementation. 1 Protocol development was driven by the 
program logic model and the following five research questions: 

1 . How was the Diplomas Now model implemented in the case-study schools? 

2. What factors facilitated implementation of the model and what factors hindered it? 

3. What are the perceived benefits of the Diplomas Now model in the case-study 
schools? 

4. What were the drawbacks to implementing the Diplomas Now model? 

5. What lessons can be learned from the implementation of the Diplomas Now model 
in the case-study schools? 

Interview and focus-group protocols contained similar sets of open-ended questions tai- 
lored to each participant group (enabling role-specific items) and focused on topics including 
implementation, collaborative activities, perceived impact, sustainability, and lessons for the 
future. 


Data Collection 

To capture participants’ experiences during the first year of implementation, case-study 
data-collection activities were conducted at three schools in spring 2012 and four schools in 
spring 2013. Participating schools represented four districts across the United States. Two to 
three research team members visited each school for three to five days to conduct interviews, 
facilitate focus groups, and observe program activities. Individual interviews — approximately 
60 minutes in length — were conducted with school-based staff members (for example, school 
transfonnation facilitators, City Year program managers and team leaders, Communities In 
Schools site coordinators, school administrators, school counselors, and instructional coaches) 
and district-based staff members (for example, instructional facilitators, Diplomas Now field 
managers, Diplomas Now Implementation Support Team representatives, school and student 
support services facilitators, and school district leaders). Focus groups with parents, students, 
teachers, and City Year corps members lasted between 60 and 90 minutes and ranged from 2 to 
1 1 participants per group (see Table C.2 for the qualitative data collection matrix). Interviews 
and focus groups were digitally recorded; research team members also took notes. During their 


'Merriam (1998). 
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visits research team members also observed after-school programs, tutoring sessions, Early 
Warning Indicator meetings, and other Diplomas Now program activities; these observations 
ranged from 17 to 61 minutes in length. Following each site visit, the research team completed 
site-visit summaries to capture their overall impressions and main takeaways from the visit. All 
interview and focus-group recordings were transcribed verbatim. 

Coding Processes and Procedures 

Transcripts were stored, managed, and analyzed using the qualitative data analysis pro- 
gram Atlas.ti. This software was selected based on the research team’s previous experience 
using it and because it can help researchers easily draw insights from different participant 
groups and data sources. Transcripts were segmented into “quotations,” or units of data that 
were relevant and could stand alone. 2 Not all sections of transcripts were assigned codes, and 
some quotations received multiple or overlapping codes. Consistent with best practice, the 
research team looked for disconfirming evidence (or negative cases), rather than solely focusing 
on exemplary success stories, by using codes such as “nonexistent activity” and “absent 
element.” 

The inductive process of coding the interview and focus-group data was iterative and 
continuously driven both by the research purpose and by the data, a recommended analysis 
practice. 3 The research team developed an initial set of codes following a review of the program 
logic model and other program materials. Codes and their corresponding definitions were 
documented in a code book and the research team reviewed the code book collectively to 
facilitate shared understanding prior to coding. 

The research team conducted two phases of coding. (See Box C.l for the coding 
framework.) The first phase included an initial content analysis, whereby codes, established a 
priori, were applied to data. 4 Each transcript was independently analyzed by two research team 
members. To ensure inter-rater reliability, the team resolved any discrepant analyses during 
weekly consensus-building meetings. These regular meetings for peer debriefing also served as 
an opportunity to confirm that processes were being applied consistently, provide feedback on 
trends in coding (for example, on the over- or underemphasis of particular codes), clarify or 
revise the operational definitions of codes, organize codes into logical groups, and determine the 
need for additional codes. Coding decisions were discussed and analyzed until the team mem- 
bers reached agreement and clarified inconsistencies. In the second phase of coding, research 


2 Lincoln and Cuba (1985). 

3 Merriam (2009). 

4 Miles and Huberman (1994); Saldana (2009); Patton (2002). 
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Box C.1 


Coding Framework for Interviews and Focus Groups 

Activity type. Program inputs and activities specific to Diplomas Now. 

Background information. Introductory and background information on interview/focus- 
group participants. 

Implementation challenges. Activities/practices identified as challenging to implement or 
barriers that reportedly made implementation a challenge. 

Implementation facilitators. Activities/practices identified as effectively implemented or 
factors that reportedly made implementation successful. 

Collaboration. Types of interactions to establish and capitalize on relationships (for example, 
formal meetings, informal communication, or planning) among key players such as Diplomas 
Now school-based staff members, administrators, and teachers. 

Perceived program impact. Effects that participants attributed to Diplomas Now program- 
specific activities/inputs and details about whom or what was affected (for example, student 
behavior, student attendance, student course performance, or school climate). 

Lessons learned. Things respondents would do differently or recommendations from the first 
year of implementation (for example, lessons, quotes, examples, or stories). 

School context. Information about the setting in which Diplomas Now is implemented (for 
example, regarding resources, school characteristics, or staff stability). 


team members conducted ad hoc analysis to identify additional themes not previously identified 
using a priori codes. This final review was conducted to ensure the application of ad hoc codes 
to all transcripts. 


Data Analysis Processes and Procedures 

After completing two phases of coding, the team analyzed the frequency by which each code 
was applied to transcripts, or its “groundedness,” to determine which codes should be aggregat- 
ed, disaggregated, or deleted. In other words, codes with no or few quotations (low grounded- 
ness) were merged with other codes and codes with high groundedness were further analyzed to 
consider whether they needed to be broken apart into more specific codes. This process looked 
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for correspondences among codes, and helped achieve a manageable number of codes based on 
frequency, uniqueness, and importance . 5 

Diplomas Now activities described during interviews and focus groups were re- 
classified according to the inputs used in the program logic model (for example, after-school 
programs organized by City Year were reclassified as “Student Support Services”). Another 
step in the analysis was to aggregate collaboration codes to examine the various interactions 
occurring over the course of Diplomas Now implementation. Co-occurrence tables were used to 
compare the frequency with which each input and collaborative interaction was coded as an 
implementation facilitator, implementation challenge, lesson learned, or perceived program 
impact. Narrative summaries were used as a data-reduction technique and included an addition- 
al process to group quotations for further inquiry based on commonly co-occurring themes . 6 
This allowed the research team to fully understand emerging themes based on the data, and to 
pull direct quotations or vignettes to illustrate relevant concepts. 


Detailed Findings from the Qualitative Research 


5 Creswell (1998); Merriam (1998). 

6 Seidman (2006); Maxwell (1995). 
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Diplomas Now 
Table C.2 


First-Year Interview and Focus-Group Findings, 

Numbers of Sessions in Which Diplomas Now Program Inputs Were Discussed, 
by Implementation Themes, Respondent Type, and School Type 



Respondent Type 


School Type 

Implementation Themes 

Number of 
Diplomas Now 
Staff Sessions 

Number of 
School/District 
Staff Sessions 

Number of 
Parent/Student 
Sessions 

Number of 
Middle 
Sessions 

Number of 
High School 
Sessions 

Pillar I. Teacher Teams and Small Learning Communities 
Strong Learning Environments 3 

Identified as effectively implemented 

5 

5 

1 

0 

11 

Identified as challenging to implement 

3 

2 

0 

0 

4 

Perceived positive impact on school or students 

0 

2 

1 

0 

3 

Lessons learned for future implementation 

4 

1 

0 

1 

4 

Pillar II. Curriculum and Instruction with Professional Development 
Curriculum for College Readiness 

Identified as effectively implemented 

2 

5 

1 

0 

8 

Identified as challenging to implement 

8 

5 

0 

8 

5 

Perceived positive impact on school or students 

0 

2 

0 

0 

2 

Lessons learned for future implementation 

3 

6 

0 

4 

5 

Professional Development and Peer Coaching a b 

Identified as effectively implemented 

8 

3 

0 

8 

3 

Identified as challenging to implement 

13 

6 

0 

10 

8 

Perceived positive impact on school or students 

0 

1 

0 

0 

1 

Lessons learned for future implementation 

19 

11 

0 

12 

17 


(continued) 




Table C.2 (continued) 



Respondent Type 


School Type 

Implementation Themes 

Number of 
Diplomas Now 
Staff Sessions 

Number of 
School/District 
Staff Sessions 

Number of 
Parent/Student 
Sessions 

Number of 
Middle 
Sessions 

Number of 
High School 
Sessions 

Pillar III. Tiered Student Supports 
Tiered Intervention Model 3 

Identified as effectively implemented 

14 

11 

0 

7 

18 

Identified as challenging to implement 

10 

7 

0 

10 

6 

Perceived positive impact on school or students 

2 

5 

0 

1 

6 

Lessons learned for future implementation 

9 

6 

0 

4 

11 

Student Supports 3 

Identified as effectively implemented 

13 

16 

4 

17 

15 

Identified as challenging to implement 

11 

10 

0 

13 

7 

Perceived positive impact on school or students 

7 

4 

1 

6 

5 

Lessons learned for future implementation 

6 

4 

1 

6 

4 

Student Case Management 3 

Identified as effectively implemented 

3 

11 

2 

7 

8 

Identified as challenging to implement 

2 

2 

1 

1 

3 

Perceived positive impact on school or students 

0 

2 

0 

0 

1 

Lessons learned for future implementation 

2 

3 

0 

3 

1 

Pillar IV. Can-Do Culture and Climate 
Integrated On-Site Support 3 

Identified as effectively implemented 

11 

9 

0 

10 

10 

Identified as challenging to implement 

7 

5 

0 

8 

3 

Perceived positive impact on school or students 

3 

1 

0 

2 

1 

Lessons learned for future implementation 

5 

5 

0 

5 

4 

Family and Community Involvement 

Identified as effectively implemented 

7 

4 

1 

5 

7 

Identified as challenging to implement 

3 

2 

2 

4 

3 

Perceived positive impact on school or students 

6 

2 

1 

6 

3 

Lessons learned for future implementation 

1 

2 

2 

2 

3 
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Table C.2 (continued) 


SOURCES: Diplomas Now program staff, school, and district interviews and focus groups, spring 2012 and 2013. 

NOTES: Data were collected at seven Diplomas Now schools representing four districts. Data-collection activities included 49 interviews and 28 focus 
groups with a total of 173 participants (25 participants per school, on average). 

This table reflects the number of interviews and focus groups in which each theme was discussed. Each interview and focus group was only counted 
once regardless of the number of times the theme was discussed during the interview or focus group session. The total number of district/school staff 
participants is higher than the sum of middle and high school participants as one interview was conducted with a district-level staff member who 
oversaw both DN middle schools and DN high schools, and who was not included in either separate count. 

Rows labeled "identified as effectively implemented" refer to instances in which interview/focus-group respondents reported that a Diplomas Now 
program input either was necessary to the success of the Diplomas Now model or was executed effectively. Rows labeled "identified as challenging to 
implement" refer to instances in which interview/focus-group respondents reported that a Diplomas Now program input was difficult to implement, did 
not work well, or was a barrier to the effective implementation of the Diplomas Now model. Rows labeled "perceived positive impact on school or 
students" refer to instances in which interview/focus-group respondents described changes in school climate, student attendance, student 
behavior/discipline, or student course performance that they attributed to the implementation of a particular program input. Rows labeled "lessons 
learned for future implementation" refer to instances in which interview/focus-group respondents talked about things they would have done differently, 
provided recommendations for other schools implementing the Diplomas Now model, described features unique to the first year of implementation, or 
described implications for sustaining Diplomas Now. 

indicates model inputs designated as critical to the Diplomas Now model. 

b While the inputs listed in the Diplomas Now logic model distinguish between "program staff professional development" and "professional 
development/peer coaching" for school staff members, the year-one case study analysis plan did not make this distinction and all types of training for 
both school and program staff members were assigned the same code. Future reports will make a distinction between training received by program staff 
members and training received by school staff members so that the analysis plan is parallel to inputs presented in the logic model. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school refonns, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



